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CHAPTER  I 
INTRODUCTION 

The  Production  Readiness  Review  (PRR)  is  an  Air 
Force  procurement  concept  that  evolved  in  the  early  1970s 
to  meet  the  demands  of  the  dynamic  weapons  acquisition 
process.  A PRR  is  a government  analysis  of  a contractor's 
readiness  to  transition  from  the  full-scale  development 
phase  into  the  production  phase  of  the  weapons  acquisi- 
tion process.  The  contractor  may  have  produced  some 
developmental  hardware,  but  the  question  addressed  by  the 
PRR  is  whether  or  not  the  contractor  is  ready  to  produce 
\ the  required  quantity  of  production  units  efficiently 

and  economically  (41:3-4).  Air  Force  Systems  Command 
(AFSC)  defines  the  Production  Readiness  Review  (PRR)  con- 
cept as  follows: 

The  PRR  is  a formal  inspection  to  determine 
whether  (1)  a system  or  equipment  under  development 
is  ready  for  efficient  and  economical  quantity  pro- 
duction; (2)  all  important  production  engineering 
problems  encountered  during  development  have  been 
resolved;  and  (3)  the  contractor  has  accomplished 
adequate  planning  for  the  production  phase  [42:1-2). 

A PRR  is  required  for  all  major  Air  Force  acqui- 
sition  programs  having  an  estimated  Research,  Development, 
Test,  and  Evaluation  ( RDT&E)  cost  exceeding  $50  million  or 
an  estimated  production  cost  exceeding  $200  million.  A 
PRR  may  be  conducted  on  any  other  development  or  production 
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program  if  the  Air  Force  Program  Manager  determines  that 
a PRR  is  necessary  (42:1). 

Statement  of  the  Problem 

Uncertainty  permeates  the  entire  weapons  acquisi- 
tion process.  To  help  cope  with  the  uncertainty  in  the 
weapons  acquisition  process,  the  Defense  System  Acquisi- 
tion Review  Council  (DSARC)  was  established.  The  DSARC's 
job  is  to  review  a military  program's  status  and  make  a 
recommendation  to  the  Secretary  of  Defense  on  the  pro- 
gram's readiness  to  transition  from  one  phase  into  the 
succeeding  phase  of  the  weapons  acquisition  process 
(Figure  1,  page  3).  The  Air  Force  is  using  the  Produc- 
tion Readiness  Review  (PRR)  as  a technique  to  prepare  for 
the  DSARC  III  review  (Figure  1,  page  3)  held  prior  to  the 
production  phase  (42:5). 

Air  Force  Systems  Command  (AFSC)  published  a regu- 
lation to  provide  general  guidance  for  conducting  a Pro- 
duction Readiness  Review  (42:1).  The  regulation  does  not 
provide  for  coordinated  PRR  planning  efforts,  standardized 
PRR  implementation,  or  feedback  of  lessons  learned  in 
previous  PRRs.  Further,  there  is  no  centralized  data 
source  to  which  Air  Force  system  program  offices  can  go 
to  receive  expert  assistance  on  how  to  plan  and  implement 
their  particular  PRR  program  (49).  As  a result,  completely 
different  PRR  approaches  were  taken  on  existing  major 
weapon  system  programs  (15;  20;  45).  Other  major  system 


program  offices  are  currently  faced  with  the  problem  of 
creating  their  own  PRR  programs  (6;  10). 

Because  of  the  lack  of  centralized  assistance 
for  planning  and  implementing  PRR  programs,  the  lack  of 
standardized  PRR  procedures,  and  the  lack  of  coordinated 
PRR  planning  efforts,  major  Air  Force  weapon  system  pro- 
grams may  be  conducting  the  Production  Readiness  Reviews 
inefficiently.  A need  exists  to  determine  if  a standard 
PRR  approach  can  be  used  by  Air  Force  system  program 
offices  for  planning  and  implementing  future  PRR  programs 
in  support  of  the  DSARC  III  decision-making  process. 

Justification  for  Research 

Relevance  to  Logistics 
Management  Environment 

According  to  a RAND  Corporation  study  (3:10-12), 
since  the  advent  of  the  atomic  age,  the  emphasis  on  weapon 
systems  acquisition  has  shifted  from  one  of  quantity  to 
one  of  quality.  The  requirement  for  massive  forces 
designed  to  deliver  large  quantities  of  explosives  has 
been  supplanted  with  highly  complex  and  sophisticated 
weapon  systems.  The  quality  emphasis  has  not  only 
resulted  in  changes  in  design  philosophy  but  has  also 
created  many  problems  in  procurement  management.  "It  is  a 
systems  complexity  that  entangles  the  development,  pro- 
duction, and  maintenance  problems  together,  creating 
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the  knottiest  problem  of  procurement  management  [3:14]." 

It  is  this  complexity  and  entanglement  that  have  brought 
about  the  establishment  of  project  type  organizations 
for  managing  the  procurement  of  the  major  weapon  systems 
(3:10-12) . 

Cost  growth  and  large  cost  overruns  on  many  mili- 
tary programs  in  recent  years  have  generated  severe 
criticisms  in  the  public  press  and  in  Congress  (23:28). 
While  inflation  and  an  expanding  technology  are  increasing 
the  costs  of  weapon  systems,  the  growing  demand  for  pub- 
lic funds  to  support  other  government  projects  has  resulted 
in  a lesser  proportion  of  the  federal  budget  available  for 
Department  of  Defense  programs  (Figures  2 and  3,  page  6). 
McQuinn  (23:28)  stated  that  the  military  procurement  envi- 
ronment has  changed  significantly.  There  is  less  money 
to  spend,  and  the  funds  available  must  be  used  more 
efficiently.  Industry  and  government  must  work  together 
to  improve  the  efficiency  and  economy  in  the  procurement 
of  military  goods  and  services  (23:28). 

According  to  Fitzgerald  (8:8),  the  cost  growth  in 
military  procurement  can  partly  be  attributed  to  the  manu- 
facturing processes  and  controls  currently  being  used 
in  the  aerospace  industry: 

Production  control  techniques  which  had  been 
carefully  evolved  and  refined  over  a period  of  sixty 
to  seventy  years  were  effectively  abandoned.  Orders 
for  parts  to  be  made  in  the  factories  would  be 
released  and  immediately  lost.  Hordes  of  "expediters" 
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or  "parts  chasers"  would  then  be  unleashed  to  find  the 
orders  and  shepherd  the  parts  through  the  manufacturing 
process.  By  dint  of  much  scurrying  about,  the  parts 
chasers  would  round  up  enough  parts  to  assemble  a 
missile  from  time  to  time  [8:8]. 

This  is  a dramatization  of  problems  in  an  industry 

that  has  been  plagued  with  waste  and  inefficiency.  To 

help  counteract  this  inefficiency  and  obtain  the  needed 

military  products  economically,  the  Air  Force  created 

the  PRR  concept  (42:1). 

The  cost  growth  problem  in  military  procurement 
has  been  investigated  by  the  General  Accounting  Office 
(GAO)  . 


The  General  Accounting  Office  report  cites 
increased  costs  of  $31.5  billion  in  DOD  acquisition 
of  45  major  systems,  or  a 39  percent  increase  over 
planning  estimates  and  a 20  percent  increase  over 
estimates  made  during  the  development  phase.  Among 
the  systems  cited  as  having  major  cost  overruns 
were  the  C-5A  cargo  plane,  the  F-lll  and  F-14  air- 
craft, the  M-60  Army  tank,  Poseidon  submarines, 
and  Minuteman  missiles  .... 


Causes  of  cost  changes,  according  to  the  GAO  report, 
can  be  attributed  to  inflation  (30  percent) , estimating 
errors  (25  percent) , and  changes  in  requirements 
ordered  by  the  military  (45  percent) . The  last  cause, 
which  involves  revisions  in  specifications,  time  sched- 
ules, and  quantities,  results  from  unrealistic  per- 
formance targets  at  the  outset  and  from  "concurrency," 
that  is,  beginning  production  before  full-scale 
development  and  testing  have  been  completed  [2:22-23], 

Belden  (2:22-24)  advocated  that  top  defense  managers  must 
be  prepared  to  respond  to  all  inquiries  requesting  expla- 
nations of  unpredicted  cost  escalations  in  military  pro- 
curement. In  some  cases,  unexpected  developments  will 
escalate  costs  and  may  be  excusable.  But  the  Congress 
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and  the  public  will  not  and  should  not  be  expected  to 
tolerate  cost  escalations  clearly  attributable  to  poor 
management  or  other  avoidable  procurement  mistakes  (2:24). 

Hagen  (14:1-17)  stated  that  the  acquisition  of 
major  weapon  systems  is  designed  to  follow  a systematic 
flow  through  well-defined  phases.  These  phases  are  con- 
ceptual, validation,  full-scale  development,  production, 
and  deployment  (Figure  1,  page  3) . A major  step  was 
taken  in  1969  to  achieve  better  coordination  of  the  various 
phases  of  the  weapons  acquisition  process.  The  Defense 
System  Acquisition  Review  Council  ( DS ARC ) was  formed  to 
evaluate  and  review  each  major  weapon  system  at  three 
critical  junctures — DSARC  I before  transitioning  into 
the  validation  phase,  DSARC  II  prior  to  moving  into 
the  full-scale  development  phase,  and  DSARC  III  before 
moving  into  the  production  phase  (14:12). 

The  purpose  of  DSARC  III  is  to  provide  a recom- 
mendation to  the  Secretary  of  Defense  on  moving  a major 
weapon  system  into  production  (Figure  1,  page  3)  (26:4). 

The  DSARC  III  must  confirm,  among  other  points, 

. . that  a practical  engineering  design,  with  adequate 
consideration  of  production  and  logistics  problems,  is 
complete  [36:4]."  DSARC  III  serves  as  the  basis  for  the 
decision  on  whether  or  not  a weapon  system  will  be 
produced  for  deployment.  DSARC  III  meetings  are  held 
when  the  military  department  determines  that  engineering 
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and  operational  systems  development  and  testing  have  been 
substantially  completed,  all  major  development  problems 
have  been  resolved,  and  the  weapon  system  is  ready  to 
transition  into  production  (26:4).  The  procuring  mili- 
tary department  must  supply  information  addressing  the 
following  areas  to  support  DSARC  III  in  the  decision- 
making process:  program  background  and  objectives,  tech- 

nical assessment  (status  of  development  and  production 
engineering),  production  and  procurement,  assessment, 
schedules,  costs,  program  management,  other  (e.g.,  com- 
pliance with  Equal  Employment  Opportunity  Act,  Armed 
Services  Procurement  Act,  etc.),  and  source  selection 
decisions  (14:12). 

In  May,  1970,  Deputy  Secretary  of  Defense  David 
Packard  directed  the  services  to  improve  their  management 
procedures  associated  with  major  systems  acquisition. 

The  prime  objective  of  the  new  policy  guidance  is 
to  enable  the  Services  to  improve  their  management  of 
programs.  Improvement  in  the  execution  of  the  programs 
will  be  made  to  the  extent  the  Services  are  willing  and 
able  to  improve  their  management  practices.  The 
Services  have  the  responsibility  to  get  the  job  done. 

It  is  imperative  that  they  do  the  job  better  in  the 
future  than  it  has  been  done  in  the  past  . . . [26:1]. 


The  start  up  of  production  must  be  scheduled  to 
minimize  financial  commitments  until  it  has  been 
demonstrated  that  all  major  development  problems  have 
been  resolved  [26:4]. 
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Utility  of  PRR  Within 
Logistics  Management- 
Environment 

Critical  Production  Time  Period.  The  most  critical 
time  period  in  terms  of  the  rate  of  expenditure  of  funds  is 
the  production  phase  of  the  weapons  acquisition  process 
(Figure  4,  page  11).  The  technical  uncertainty  level 
should  have  been  reduced  by  this  time,  and  the  major  obsta- 
cle for  the  system  program  office  is  to  convince  DSARC  and 
Congress  that  the  program  is  ready  to  transition  from  the 
full-scale  development  phase  into  the  production  phase  of 
the  acquisition  process  (4) . One  estimate  of  a typical  Air 
Force  program's  costs,  excluding  operation  and  support 
costs,  is  that  about  33  percent  of  a program's  funds  are 
committed  for  research  and  development,  and  the  other  67 
percent  are  spent  in  the  production  phase  of  the  weapons 
acquisition  process  (49)  . Both  the  greatest  cost  over- 
runs and  the  greatest  potential  savings  can  occur  in  the 
production  phase  (49) . 

The  large  number  of  taxpayers'  dollars  which  are 
expended  in  the  production  phase  of  major  weapon  system 
contracts  justifies  the  need  for  research  in  the  PRR 
area  (31).  According  to  Rogers  (31),  the  Department  of 
Defense  production  expenditures  for  fiscal  year  1973  were 
approximately  2.2  times  that  spent  for  Research,  Develop- 
ment, Test,  and  Evaluation  (RDT&E)  activities.  The 
average  amount  of  Air  Force  Systems  Command  production 
dollars  spent  in  fiscal  years  1971,  1972,  and  1973  was 
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Fig.  4.  The  Flow  of  Expenditures  in  a Typical  Weapons 
Program  (28:310) 
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1.7  times  greater  than  the  expenditure  for  RDT&E  activi- 
ties (31) . With  the  large  cost  overruns  that  have  been 
experienced  in  military  procurement  programs  in  recent 
years,  there  is  a need  to  reduce  the  uncertainty  level 
for  a weapon  system  prior  to  production  authorization. 

. . . government  should  strive  to  minimize  the 
total  sacrifice  made  by  taxpayers  . . . [35:20]. 

One  objective  of  government  intervention  is  to 
enhance  the  conditions  under  which  economic  activity 
takes  place  so  that  the  realities  of  production  match 
production  possibilities  [35:30]. 

Previous  Production  Phase  Problems.  The  preferred 
method  of  managing  the  procurement  program  of  a major 
weapon  system  is  to  prevent  potential  problems  rather  than 
react  to  problems  after  they  occur  (16:537-538).  Unfor- 
tunately, reactive  management  has  often  occurred  in  the 
production  management  of  some  of  our  major  weapon  sys- 
tems (48:154-166).  One  primary  reason  for  this  reactive 
management  has  been  the  lack  of  sufficient  information  by 
which  to  make  decisions  on  whether  or  not  to  continue 
funding  a major  weapon  system  program.  This  lack  of  infor- 
mation commonly  occurs  when  a production  phase  of  a con- 
tract is  started  before  the  full-scale  development  phase 
has  identified  potential  problem  areas  (48:154-166).  The 
concurrency  problem  surfaced  in  the  procurement  of  the 
C-5A  and  F-14  weapon  systems: 

The  C-5A  and  F-14  aircraft  programs  demonstrated 
the  problems  that  can  result  from  lack  of  early 
development  tests.  The  Government  has  paid,  and  is 


continuing  to  pay,  heavy  "penalties"  for  authorizing 
extensive  contractor  production  before  completion  of 
critical  test  phases.  The  overlap  of  production, 
development,  and  related  testing  is  referred  to  as 
"concurrency."  Illustrations  of  the  impact  of  con- 
currency on  use  of  test  results  follow.  Production 
of  all  C-5A  aircraft  was  started  and  more  than  half 
of  them  were  produced  before  completion  of  ground 
testing  and  initial  R&D  flight  tests.  Operational 
aircraft  later  accepted  by  the  Air  Force  had  47  major 
deficiencies  of  which  14  impaired  the  aircraft's 
capabilities  to  perform  all  or  part  of  its  six 
missions  [48:158]. 

The  need  to  evaluate  a weapon  system's  readiness 
to  transition  from  development  to  production  is  further 
discussed  by  David  Packard: 

There  has  been  real  waste  of  both  time  and  money 
in  almost  every  program  in  which  production  was 
started  before  development  and  testing  was  completed. 
That  includes  almost  every  program  [27:4]. 

The  acknowledged  production  phase  problems  indicate  the 

need  for  assessing  a program's  readiness  to  transition 

from  development  into  production  before  authorizing  a 

contractor  to  proceed  into  the  production  phase  of  the 

weapons  acquisition  process. 


CHAPTER  II 


BACKGROUND 


Weapons  Acquisition 
Process  Environment 

A continuum  of  uncertainty  exists  throughout  the 
weapons  acquisition  process  (Figure  5,  page  15).  The 
levels  of  uncertainty  can  be  classified  as  anticipated 
unknowns,  unanticipated  unknowns,  and  knowns  (48:148). 

The  highest  degree  of  uncertainty  exists  in  the  very 
early  phases  of  the  weapons  acquisition  process  (con- 
ceptual and  validation  phases) . In  these  early  phases 
the  technical,  cost,  and  schedule  unknowns  are  the 
greatest.  As  a program  proceeds  through  the  weapons 
acquisition  process,  more  program  information  is  obtained 
which  reduces  the  level  of  uncertainty  (development 
and  production  phases) . In  these  latter  phases  some 
unanticipated  unknowns  may  be  encountered,  but  the 
level  of  knowns  in  the  uncertainty  continuum  has  increased 
(48:148).  As  a weapon  system  moves  into  the  deployment 
phase,  complete  cost,  schedule,  and  technical  information 
should  be  available  for  decision  making  (22:12). 

Statement  of  Key  Concepts 

The  Air  Force  is  currently  using  various  methods 

such  as  pre-award  surveys.  Manufacturing  Management/ 

14 
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Production  Capability  Reviews,  and  Production  Readiness 
Reviews  to  reduce  the  uncertainty  level  in  the  production 
management  area  throughout  the  weapons  acquisition  pro- 
cess (5).  Campbell  (5)  stated  that  some  formal  techniques 
are  used  prior  to  contract  award,  and  others  are  used  at 
selected  points  in  the  weapons  acquisition  process.  The 
formal  techniques  are  used  to  evaluate  a contractor's 
current  capability,  potential  capability  to  perform  on 
a contract,  or  state  of  readiness  to  transition  from  one 
phase  to  the  succeeding  phase  of  the  weapons  acquisition 
process.  In  addition  to  the  continuous  contractor  evalu- 
ation programs  conducted  by  various  government  contract 
management  organizations,  the  formal  techniques  which 
are  used  to  evaluate  the  manufacturing  or  production  capa- 
bility of  a contractor  can  be  categorized  into  two  time 
periods — source  selection  process  and  preproduction  phase 
evaluation  (5). 

Source  Selection  Process.  Uncertainties  must  be 
addressed  in  the  early  stages  of  the  weapons  acquisition 
process  (21)  . One  management  and  decision-making  mech- 
anism designed  to  cope  with  the  early  confrontation  of 
risk  is  the  source  selection  process  (22:20).  The  prime 
objectives  of  the  formalized  source  selection  process 
are  to  ensure  that: 

1.  Proposals  are  solicited  and  evaluated  and  the 
selection  decision  made  with  minimum  complexity  and 
maximum  efficiency. 
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2.  An  impartial,  equitable,  economic  but  thorough 
evaluation  of  each  offeror's  proposal  and  related 
capabilities  is  accomplished  so  that  the  evaluation 
results  can  be  properly  compared  and  presented  to  the 
Source  Selection  Authority.  The  intent  is  to  allow 
the  source  selection  decision  maker  to  objectively 
select  the  optimum  proposal  consistent  with  govern- 
ment requirements. 

3.  Solicitations  and  contracts  are  appropriately 
structured  to  equitably  distribute  technical,  finan- 
cial, and  economic  or  business  risks,  consistent  with 
the  technical  requirements  and  needs  of  the  specific 
program. 

4.  The  source  selection  process  is  sufficiently 
flexible  to  accommodate  the  objectives  of  the  pro- 
curement strategy  and  that  the  source  selection  pro- 
cedures are  compatible  with  program  requirements, 
risks, and  conditions  [22:20]. 

According  to  Mastromonico  (22:21),  the  final  evaluation  and 
overall  conclusions  on  the  capabilities  of  a contractor 
are  termed  as  strengths,  weaknesses,  and  risks.  To 
analyze  these  factors  in  the  source  selection  process, 
realistic  and  measurable  standards  are  developed.  The 
standards  are  designed  to  represent  a balance  among  sys- 
tem performance,  schedule,  and  cost,  to  allow  trade  offs 
and  comparisons  between  alternative  proposals  (22:21). 

In  the  source  selection  process,  the  government 
has  established  minimum  standards  of  responsibility  to 
be  met  by  competing  prime  contractors  and  their  respective 
subcontractors  (38:K-303.2) . 

Generally  surveys  include  determinations  as  to 
whether  the  prospective  contractor  (1)  has  or  can 
obtain  adequate  financial  resources,  (2)  is  able  to 
meet  delivery  dates,  taking  into  consideration  all 
existing  business  commitments,  (3)  has  a satisfactory 
record  of  performance  and  integrity,  (4)  has  adequate 
facilities  and  technical  capability,  (5)  has  an  ade- 
quate quality  assurance  program,  and  (6)  is  qualified  and 
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eligible  under  applicable  laws  and  regulations  to 
receive  an  award  [23:13]. 

Three  formal  techniques  are  currently  being  used  by  Air 
Force  procurement  to  evaluate  the  contractor's  capability 
in  the  source  selection  process — pre-award  surveys.  Manu- 
facturing Management/Production  Capability  Reviews  (MM/PCR) 
and  "should  cost"  teams  (5) . 

1.  Pre-Award.  Survey.  Pre-award  surveys  (PAS)  on 
potential  contractors  are  requested  by  the  procuring  con- 
tracting officer  (37:1-903.2).  According  to  McQuinn 
(23:13-14),  the  PAS  is  a technique  to  obtain  some  informa- 
tion about  potential  contractors'  capabilities  in  order 
to  reduce  the  initial  uncertainty.  The  PAS  is  defined 
as  an  evaluation  by  a government  contract  administration 
office  of  a prospective  contractor's  capability  to  per- 
form under  the  terms  of  the  proposed  contract.  In 
practice  the  pre-award  survey  is  normally  limited  to  an 
assessment  of  a contractor's  capability,  workload,  and 
historical  contract  performance  (23:13-14).  A typical 
pre-award  survey  team  may  include  the  following  func- 
tional experts:  engineer  (mechanical,  electrical, 

general) , industrial  specialist,  financial  auditor,  quality 
assurance  specialist,  specialized  safety  and  flight  opera- 
tions officer,  industrial  labor  relations  officer,  and 
industrial  security  officer  (23:14). 
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In  requesting  the  pre-award  survey,  the  purchasing 
office  usually  specifies  the  scope  and  determines  which 
factors  should  receive  special  emphasis  (23:13).  The 
procuring  contracting  officer  uses  the  pre-award  survey 
report,  in  conjunction  with  other  available  information, 
to  determine  whether  the  prospective  contractor  should  be 
awarded  the  contract  (37:1-903.2).  The  pre-award  survey 
is  used  to  help  reduce  the  uncertainty  concerning  the 
contractor's  capability  to  produce  both  before  initial 
contract  award  and/or  prior  to  any  subsequent  follow-on 
contract  awards  to  the  same  contractor  (23:13-14). 

2.  Manufacturing  Management/Production  Capability 
Review  (MM/PCR).  The  MM/PCR  is  an  effort  on  the  part  of 
AFSC's  Aeronautical  Systems  Division  (ASD)  to  supplement 
and  improve  upon  the  current  pre-award  survey  as  a source 
selection  evaluation  technique  (5) . The  MM/PCR  is  not  an 
attempt  to  replace  the  pre-award  survey;  the  MM/PCR  is 
recommended  as  a replacement  for  the  current  production 
management  portion  of  the  pre-award  survey  for  major  weapon 
systems  (44:1). 

The  MM/PCR  is  performed  on  each  contractor  being 
considered  for  award  as  part  of  the  source  selection 
process.  The  MM/PCR  was  used  in  the  source  selection 
activities  for  the  recent  Air  Combat  Fighter  (ACF)  compe- 
tition between  General  Dynamics  and  Northrop  (5) . 

The  purpose  of  a MM/PCR  is  to  determine  if  a con- 
tractor has,  or  can  obtain  in  a timely  fashion,  the 
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required  production  capability  to  perform  all  existing 
and  planned  activities  including  the  proposed  pro- 
curement plus  any  anticipated  follow-on  production 
requirements  [44:1] . 

A team  approach  is  used  in  performing  the  MM/PCR 
(5).  The  MM/PCR  team  participants  are  selected  from 
several  government  activities  including  the  system  program 
office,  the  system  division  production  staff,  contract 
management  activities , and  supplementary  personnel  from 
other  AFSC  divisions  and  technical  organizations  as  required 
(44:3).  The  MM/PCR  is  used  to  analyze  a contractor's 

\ 

manufacturing  proposal  at  the  operational  level. 

The  report (s)  on  initial  MM/PCR(s)  conducted  during 
source  selection — pre-award  survey  phase  establish (es) 
a baseline (s)  (point  of  departure)  for  subsequent 
Production  Readiness  Reviews.  . . . [44:1-3]. 

3.  Should  Coat.  A third  technique  currently 
being  used  to  reduce  uncertainty  in  the  source  selection 
process  is  "should  cost."  The  technique  has  as  its  goal 
the  detailed  evaluation  of  a contractor's  cost  and  per- 
formance data  in  order  to  establish  a government  position 
as  to  what  the  production  of  a particular  item  should 
cost  (47:1) . 

A major  task  of  military  procurement  is  to  select 
a suitable  means  to  determine  what  constitutes  a fair  and 
reasonable  price  for  a weapon  system  (23:28).  A fair 
and  reasonable  price  is  one  that  provides  an  adequate 
profit  to  the  defense  contractor  to  keep  him  in  business, 
but  does  not  pay  for  any  wasteful  and  unnecessary  busi- 
ness practices  (23:28).  ’ 
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McQuinn  (23:29)  stated  that  traditionally  military 
procurement  has  negotiated  weapon  system  prices  based  on 
past  experience — sometimes  called  "historical  costs"  approach. 
The  approach  used  as  its  basis  the  cost  of  producing  a 
comparable  weapon  system  with  some  adjustments  for  infla- 
tion, new  technology,  and  more  stringent  performance 
requirements.  According  to  McQuinn,  Senator  William  Proxmire 
and  other  military  procurement  critics  have  denounced  the 
historical  approach.  They  argue  that  past  costs  may  have 
been  inflated  intentionally,  through  inefficiency,  or  because 
no  competition  existed.  Past  costs  then  are  misleading 
base  figures  to  analyze  future  contracts  (23:29).  The 
"should  cost"  concept  is  a technique  to  determine  what 
weapon  systems  should  cost  based  on  reasonable  efficiency 
and  economy  of  operations  (12:47). 

In  general,  "should  cost"  teams  study  the  contrac- 
tor's on-site  operations  for  an  extended  period  of  time 
reviewing  such  areas  as  industrial  engineering  standards, 
plant  layout,  production  methods,  purchasing  systems, 
accounting  systems,  and  overhead  structure.  The  "should 
cost"  team  documents  its  findings  and  quantifies  the 
effect  that  recommended  changes  would  have  on  the  weapon 
system's  final  price  using  the  assumption  of  reasonable 
economy  and  efficiency  (23:30).  The  "should  cost"  approach 
is  described  explicitly  in  military  procurement  guidelines 
(47:1)  . 


Preproduction  Phase  Evaluation.  Formalized  tech- 


niques (e.g.,  PAS,  MM/PCR,  "should  cost")  are  available 
to  reduce  the  uncertainty  in  the  early  phases  of  the 
weapons  acquisition  process.  But  in  the  latter  phases  of 
the  process,  few  information-gathering  techniques  are 
available  (40:16-23).  Some  techniques  used  to  reduce 
uncertainty  in  the  latter  phases  of  the  weapons  acquisi- 
tion process  are  discussed  below. 

1.  Production  Readiness  Revieu  (PRR).  A promi- 
nent Air  Force  study  group  noticed  an  information  gap  when 
it  came  time  to  determine  if  a contractor  was  ready  to 
transition  from  the  development  phase  to  the  production 
phase  of  the  weapons  acquisition  process  (40:16-23).  At 
this  stage  of  the  weapons  acquisition  process,  many  of  the 
major  weapon  system  acquisitions  may  be  categorized  as 
bilateral  monopolies  (21)  . The  monopsonist  is  the  govern- 
ment, and  the  monopolist  is  the  sole-source  contractor 
(18:264).  The  Air  Force  initiated  the  formal  PRR  proce- 
dure to  obtain  information  for  evaluating  a sole-source 
contractor's  readiness  to  make  the  development  to  pro- 
duction transition  (14:13). 

Air  Force  Systems  Command  Regulation  (AFSCR) 

, 84-2  (42:4)  prescribes  that  the  PRR  team  size  and  compo- 
sition be  established  based  on  the  scope  of  the  review 
effort.  Primary  PRR  team  members  normally  possess 
expertise  in  the  following  functional  areas: 
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(1)  industrial  or  production  engineering,  (2)  production 
management,  (3)  program  or  project  planning  and  production 
control,  and  (4)  manufacturing  methods,  tool  design  and 
test  requirements,  plant  layout,  etc.  (42:4).  Since  the 
procurement  process  involves  many  related  functional 
disciplines,  the  PRR  team  may  need  to  be  augmented  by 
experts  from  other  functional  areas  (14:13). 

AFSCR  84-2  (42:3)  also  states  that  the  PRR  team 
conducts  an  on-site  examination  of  the  contractor's 
working  papers,  documents,  production  methods,  processes, 
and  techniques.  To  assess  the  contractor's  readiness  to 
transition  into  production,  the  PRR  team  looks  at  existing 
data  and  procedures.  The  contractor  is  not  required  to 
prepare  any  special  data  or  reports  for  the  review  team 
(42:3).  The  PRR  team  uses  some  predetermined  criteria 
as  standards  to  conduct  an  orderly,  penetrating,  and  con- 
clusive review  (42:4-6). 

AFSCR  84-2  (42:3)  further  stipulates  that  if  a 
contractor  does  not  meet  acceptable  standards,  then  cor- 
rective actions  within  the  scope  of  the  contract  may  be 
required.  The  contractor's  weak  areas  are  identified 
in  the  PRR,  and  a schedule  for  subsequent  reviews  is 
developed.  At  the  conclusion  of  the  PRR,  the  PRR  team 
director  submits  a formal  report  to  the  Air  Force  Program 
Manager.  The  report  describes  the  contractor's  weak 
areas  and  gives  a recommendation  on  the  program's 
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production  readiness  status.  Program  Managers  and  other 
Air  Force  management  levels  use  the  final  PRR  report  to 
make  their  final  production  recommendation  to  other 
Department  of  Defense  agencies  (42:3). 

Air  Force  Systems  Command  (AFSC)  has  three  major 
product  divisions — Electronic  Systems  Division  (ESD) , 
Hanscom  AFB,  Massachusetts;  Aeronautical  Systems  Division 
(ASD) , Wright-Patterson  AFB,  Ohio;  and  Space  and  Missile 
Systems  Organization  (SAMSO) , Los  Angeles,  California  (20). 
Production  Readiness  Reviews  have  been  conducted  at  each 
of  these  three  divisions  in  some  form.  A background  review 
disclosed  the  list  of  PRRs  in  Table  1 that  have  been  com- 
pleted or  are  in  progress  (13) . 

Table  1 


Production  Readiness  Review  Programs 


AFSC  Procuring 
Division 

Number  PRRs 
Completed 
or  in 
Progress 

Air  Force 
Programs 

ESD 

1 

AWACS 

ASD 

5 

B-l 
A-10 
F-15 
F-16 
ARC  164 

SAMSO 

2 

Minuteman  Mark  XII 
Shroud 

Communications 

Satellite/TRW 

2.  Preproduction  Evaluation  Contracts . Nclan 
(25:15-16)  stated  that  one  major  problem  area  in  making 
the  transition  from  development  to  production  efficiently 
and  economically  is  the  control  or  at  least  the  identifi- 
cation of  program  impacts  that  may  result  from  engineering 
changes  after  production  go-ahead.  One  attempt  to  alle- 
viate some  of  the  problems  in  this  area  has  been  conducted 
by  United  States  Army  procurement  through  the  use  of  Pre- 
production  Evaluation  contracts  (25:15-16).  This  tech- 
nique is  an  early  attempt  to  stop  cost  escalation  that 
would  be  associated  with  engineering  changes  that  occur 
after  contract  award  (25:15). 

A persistent  procurement  problem  is  that  of  con- 
tractors "buying-in"  with  an  unusually  low  bid  and  then 
planning  for  profits  from  engineering  changes  (4).  To 
help  prevent  the  problem.  Army  procurement  requests  that 
prospective  contractors  include  an  estimate  in  their 
initial  contract  proposal  to  cover  any  anticipated  engi- 
neering changes  that  may  be  required  for  the  end  item  to 
meet  specifications.  The  Preproduction  Evaluation  con- 
tract is  an  early  attempt  to  make  the  contractor  analyze 
and  identify  the  potential  cost  escalation  factors  to 
the  procuring  activity  (25:15-16). 

3.  Department  of  Defense  Product  Engineering 
Services  Office  (PESO).  PESO  is  a Department  of  Defense 
level  organization  that  provides  independent  assessments 
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on  a military  program's  readiness  to  transition  from  the 
full-scale  development  phase  into  the  production  phase  of 
the  weapons  acquisition  process  (4).  PESO,  which  reports 
directly  to  the  Deputy  Assistant  Secretary  of  Defense 
(Installations  and  Logistics)  for  Production  Engineering 
and  Materiel  Acquisition,  conducts  independent  production 
assessment  reviews  at  contractors'  facilities.  PESO 
provides  a recommendation  to  DSARC  III  on  the  major 
procurement  programs  of  each  military  department  (4) . 

4.  Army  and  Navy  Evaluation  Techniques . The 
formalized  Production  Readiness  Review  (PRR)  concept 
is  unique  to  Air  Force  procurement.  The  Army  (34)  and 
the  Navy  (9)  do  not  have  a comparable  evaluation  system  to 
determine  a contractor's  readiness  to  transition  from 
full-scale  development  into  the  production  phase  of  the 
weapons  acquisition  process.  The  other  services'  buying 
activities  do  obtain  information  to  support  a DSARC  III 
recommendation,  but  the  review  system  is  not  formalized 
like  the  Air  Force's  PRR  program  (9;  34). 

5.  Industry  Evaluation  Techniques.  Commercial 
practice  is  not  burdened  with  many  of  the  complications 
that  surround  and  impact  military  procurement  (20)  . Cor- 
porations conducting  commercial  business  do  not  spend 
taxpayers'  money  and  are  not  restricted  by  many  of  the 
internal  and  external  factors  that  are  placed  on  military 
procurement  activities  (2:13-26).  Instead  of  meeting  a 
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DSARC  level  review  for  program  authorizations,  program 
managers  and  functional  managers  in  private  industry 
normally  have  more  freedom  in  the  management  of  commer- 
cial projects  (20) . 

According  to  England  and  Leenders  (7:456-465), 
the  factors  evaluated  by  commercial  firms  in  selecting 
acceptable  suppliers  are  progressiveness,  financial 
strength,  honesty  and  fairness,  adequate  technical  compe- 
tence, and  acceptable  production  capacity.  Information  is 
normally  gathered  from  trade  journals,  industrial  adver- 
tising, salesmen,  catalogs,  trade  directories,  samples, 
and  plant  visits  (7:456-465).  Nolan  (25:6-9)  discovered 
that  the  commercial  firm's  purchasing  decision  is  normally 
based  on  best  judgment.  After  a commercial  firm  makes  an 
initial  determination  that  another  contractor  is  able  to 
fulfill  a contract,  the  buyer  usually  does  not  question 
the  seller's  capability  to  perform  except  when  the  quality 
of  delivered  units  becomes  unacceptable  (25:6-9).  Although 
there  is  sufficient  literature  on  commercial  procurement 
practices,  no  formalized  production  readiness  evaluation 
techniques  were  discovered.  The  only  commercial  evalu- 
ation methods  are  those  that  recommend  for  the  buyer  to 
evaluate  production,  financial  strength,  and  integrity 
using  best  judgment  as  the  guide  (25:6-9). 
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Related  Research 

The  presence  of  uncertainty  within  the  weapons 
acquisition  process  is  well  recognized  (48:148).  Military 
procurement  personnel  have  a major  responsibility  to 
reduce  the  level  of  uncertainty  to  an  acceptable  level 
before  committing  taxpayers'  money  (28:v).  To  help  cope 
with  the  uncertainty  within  the  weapons  acquisition  pro- 
cess, major  weapon  system  authorizations  are  currently 
being  made  in  incremental  steps  by  using  DSARC  in  lieu 
of  the  old  single  decision  method  (26:1-5). 

The  background  review  was  directed  towards  the 
research  efforts  that  had  been  conducted  on  the  information- 
gathering processes  used  by  system  program  offices  in 
making  the  production  transition  recommendations.  Specifi- 
cally, the  background  review  concentrated  on  the  research 
projects  dealing  with  the  information-gathering  processes 
currently  being  used  to  support  the  final  DSARC  evalu- 
ation step — transition  into  the  production  phase  of  the 
weapons  acquisition  process.  The  related  research  can  be 
categorized,  in  three  basic  areas:  (1)  general  research 

indicating  the  need  for  techniques  like  the  Production 
Readiness  Review  (PRR) , (2)  unpublished  reports  that  are 
primarily  descriptive  analyses  of  the  PRR  concept,  and 
(3)  the  nature  of  the  scientific  research  on  the  various 
applications  of  the  PRR  technique. 
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General  Research.  The  majority  of  research 
efforts  that  have  indicated  the  need  for  information- 
gathering techniques  like  the  PRR  have  been  government 
study  projects  (39:63-82;  40:1;  48:85-86).  One  well-known 
study  panel  discussed  the  uncertainty  within  the  weapons 
acquisition  process  and  the  requirement  to  standardize 
control  mechanisms  within  the  procurement  process  as 
follows : 


Uncertainty  is  inherent  in  the  nature  of  the 
programs  which  involve  advances  in  technology  and  this 
uncertainty  makes  it  inevitable  that  some  degree  of 
cost  growth,  delays,  and  short-falls  in  desired 
performance  will  occur  in  some  programs.  The  fre- 
quency and  magnitude  of  such  problems  which  have  been 
experienced,  however,  surpass  significantly  those  which 
can  be  attributable  to  unavoidable  causes.  It  is  clear 
that  a substantial  portion  of  the  acquisition  problems 
must  be  attributed  to  management  deficiencies  [34:63]. 


. . . there  has  been  little  standardization  or 
reduction  in  the  number  of  management  control  systems 
contractually  applied  [34:82]. 

The  underlying  problem  in  the  acquisition  of 
major  weapon  systems  is  "the  lack  of  visibility  over  the 
key  decisions  that  control  the  purpose  and  direction  of 
system  acquisition  programs  [48:70]."  The  Congress, 
DSARC,  and  other  government  leaders  should  receive 
sufficient  information  to  make  the  "right"  program  deci- 
sions and  funding  commitments  (48:85-86).  The  need  to 
have  sufficient  information  for  the  decision-making 
process  was  discussed  in  one  report  as  follows : 

One  of  the  primary  findings  of  our  study  is  that 
too  much  is  committed  to  individual  major  systems 
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before  ideas,  needs,  designs,  and  hardware  are  tested 
and  evaluated  .... 

. . . just  prior  to  a planned  full-production 
commitment,  tests  should  be  conducted  for  the  specific 
purpose  of  making  a "go/no-go"  decision  [48:85-86], 

In  1970,  Mr.  Philip  N.  Whittaker,  the  Assistant 
Secretary  of  the  Air  Force  for  Installations  and  Logistics, 
expressed  concern  about  whether  sufficient  emphasis  was 
being  given  within  the  Air  Force  to  the  preparation  for 
and  the  management  of  the  production  phase  of  major  weapon 
acquisitions  (50:1).  An  Air  Force  study  was  conducted  to 
determine  if  the  Air  Force  had  acceptable  procedures  to 

(1)  accomplish  production  planning  during  the  full-scale 
development  phase  of  the  weapons  acquisition  process, 

(2)  formally  document  and  review  production  criteria 
prior  to  the  production  go-ahead  decision,  and  (3)  con- 
tinuously monitor  the  production  program  after  the 
production  decision  is  made  (40:1).  The  study  team  con- 
ducted comprehensive  reviews  of  Air  Force  Systems  Com- 
mand's production  management  activities  in  various  field 
organizations,  system  program  offices,  and  contractor 
plants.  One  specific  recommendation  of  the  study  group 
was  that  the  Air  Force  should  conduct  and  document  a 
formal  Production  Readiness  Review  (PRR)  prior  to  the 
production  decision  for  a major  weapon  system  (43:3). 

As  a result  of  the  study,  a PRR  requirement  was  levied 
on  all  major  weapon  system  procurements,  and  an  AFSC 
regulation  was  published  to  cover  application  of  the  PRR 
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concept  (42:1).  PRRs  are  currently  being  conducted  by 
most  of  the  production  management  organizations  throughout 
Air  Force  Systems  Command  (13) . 

Unpublished  Reports.  Only  two  unpublished  reports 
have  been  located  on  the  PRR  concept.  In  the  first 
report,  Hagen  (14:1-17)  made  a cursory  review  of  PRR  as 
part  of  an  analysis  of  the  Air  Force's  current  approach  to 
production  management.  Although  the  production  management 
function  has  been  delegated  the  responsibility  for  the  PRR, 
assistance  is  required  from  each  functional  directorate 
within  the  system  program  office.  Air  Force  Plant  Repre- 
sentative Office,  various  Air  Force  Laboratories,  and  the 
contractor's  organization.  The  PRR  must  be  coordinated 
to  include  the  results  of  engineering,  configuration,  and 
testing  outcomes  (14:13). 

In  the  second  unpublished  report,  Lockwood  stated 
that  the  PRR  is  useful  for  assessing  the  readiness  of 
a program  to  enter  the  production  phase  of  the  weapons 
acquisition  process  and  for  anticipating  future  success 
on  the  program  (19  :i).  The  Air  Force  Systems  Command's 
Aeronautical  Systems  Division  has  used  the  PRR  concept 
on  several  programs  (5).  Lockwood  (19:12-13)  stated 
that  even  though  application  of  the  PRR  concept  is  still 
in  its  infancy,  a number  of  advantages  have  been  derived 
from  conducting  PRR  programs.  First,  the  PRR  participants 
become  more  familiar  with  the  hardware  being  produced, 
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the  contractor's  management  systems,  the  functional  disci- 
plines other  than  production  management,  and  the  inter- 
actions required  to  develop  a contractor/government 
management  team.  Second,  a PRR  increases  the  probability 
of  detecting  production  problems  early  enough  to  implement 
corrective  actions.  Third,  the  PRR  concept  provides  a 
reorientation  to  insure  that  producibility 1 aspects  are 
considered  earlier  in  engineering  design  activity 
(19:12-13).  Lockwood  further  stated  that: 

A Production  Readiness  Review  that  is  considered 
with  the  objectives  of  (a)  identifying  current  problems 
and  solving  them  and  (b)  identifying  potential  problems 
and  preventing  them  will  lead  to  a decision  to  enter  a 
production  program  that  will  minimize  the  sacrifice 
by  the  taxpayer.  Objectively  conducted  in  an  open, 
cooperative  manner,  the  Production  Readiness  Review 
can  yield  mutual  benefits  to  both  buyer  and  seller 
[19:15] . 

Status  of  Scientific  Reseach.  The  Production 
Readiness  Review  was  established  to  help  reduce  uncer- 
tainty and  provide  more  control  within  the  DSARC 
decision-making  process  (42:1-5).  No  scientific  research 
was  located  that  dealt  solely  with  current  applications 
of  the  Air  Force  PRR  concept.  The  Logistics  Management 
Institute  (30:1)  and  the  RAND  Corporation  have  not  con- 
ducted any  research  in  the  area  of  the  Production  Readiness 
Review  concept.  The  Defense  Documentation  Center  and  the 

‘Producibility  means  to  design  a product  that  can 
also  be  efficiently  manufactured  (12:20-21). 


Defense  Logistics  Studies  Information  Exchange  do  not 
have  any  research  publications  pertaining  directly  to 
the  subject  of  PRR. 

Many  organizations  have  expressed  interest  in 
receiving  the  end  product  of  research  conducted  in  the 
PRR  area.  Major  organizations  interested  in  the  PRR 
research  results  include  Headquarters  AFSC  (24),  AFIT 
School  of  Systems  and  Logistics  Continuing  Education 
Division  (13) , F-16  System  Program  Office  (10)  , Airborne 
Warning  and  Control  System  Program  Office  (49)  , B-l  Sys- 
tem Program  Office  (6) , and  the  Simulator  System  Program 
Office  (11) . 


Scope  of  Research 

Since  the  PRR  requirement  was  levied  by  Air  Force 
Systems  Command  in  November,  1971,  eight  Production  Readi- 
ness Reviews  have  been  completed  or  are  currently  in 
progress  (13).  Of  these  eight  PRR  programs,  this  research 
effort  was  limited  to  three  major  weapon  system  programs 
that  have  completed  a PRR  program  and  have  received  a 
DSARC  III  decision.  By  limiting  the  research  to  the 
programs  that  have  conducted  a complete  PRR  program,  the 
entire  PRR  program  cycle  from  PRR  program  planning  to 
final  DSARC  III  review  could  be  analyzed  to  determine  the 
possible  applications  to  future  major  weapon  system  PRR 
programs . 
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Research  Objectives 

The  research  effort  had  two  objectives.  The  first 
objective  was  to  analyze  the  PRR  programs  that  have  been 
completed  on  major  Air  Force  weapon  system  programs  to 
date  to  determine  what  disparities  existed  among  the  PRR 
approaches.  The  second  objective  was  to  determine  if  a 
standard  PRR  approach  was  feasible  to  satisfy  the  Air 
Force  Systems  Command  Regulation  (AFSCR)  84-2  requirements. 

Research  Hypotheses 

1.  Air  Force  major  weapon  system  programs  utilize 
different  Production  Readiness  Review  (PRR)  program 
approaches  to  comply  with  AFSCR  84-2. 

2.  Air  Force  Systems  Command  major  weapon  system 
program  offices  can  follow  a standard  Production  Readiness 
Review  (PRR)  approach  to  comply  with  AFSCR  84-2. 


CHAPTER  III 


RESEARCH  METHODOLOGY 

The  previous  two  chapters  described  the  background 
of  the  current  production  management  situation  as  related 
to  the  Production  Readiness  Reviews  (PRR)  and  discussed 
the  need  to  reduce  uncertainty  before  transitioning  from 
the  development  phase  into  the  production  phase  of  the 
weapons  acquisition  process.  The  use  of  the  Production 
Readiness  Review  (PRR)  by  AFSC  to  help  reduce  uncertainty 
before  committing  production  funds  for  major  weapon  sys- 
tems was  also  discussed.  This  chapter  describes  the 
universe  and  population  of  interest  for  the  research, 
identifies  and  operationally  defines  the  variables  used 
in  the  data  collection  process,  and  describes  the  proce- 
dures used  in  analyzing  the  data  obtained  in  the  research. 

Universe  Description 

Since  Production  Readiness  Reviews  (PRR)  are  unique 
to  Air  Force  Systems  Command  (AFSC)  procurement  activities, 
the  universe  for  the  research  consisted  of  those  major 
weapon  system  programs  that  were  required  to  adhere  to 
AFSCR  84-2  and  which  had  an  estimated  Research,  Develop- 
ment, Test  and  Evaluation  ( RDT&E)  cost  exceeding  $50  mil- 
lion or  an  estimated  production  cost  exceeding  $200  million 

(41:1).  Two  populations  were  identified  within  the 
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universe.  Population  I consisted  of  those  major  weapon 
system  programs  within  AFSC  that  had  completed  an  entire 
PRR  program  and  had  met  a DSARC  III  review.  Population  II 
consisted  of  those  Air  Force  Systems  Command  major  weapon 

system  programs  that  are  presently  conducting  PRR  programs 

, ♦ 

or  will  be  required  to  conduct  PRR  programs  in  the  future. 

Population  of  Interest 

A census  of  Population  I was  conducted  in  the 
research.  Population  I consisted  of  three  major  weapon 
system  programs:  (1)  Airborne  Warning  and  Control  System 

(AWACS) , which  is  being  procured  by  AFSC's  Electronic 
Systems  Division, Hanscom  AFB,  Massachusetts;  (2)  F-15 
System,  which  is  being  procured  by  AFSC's  Aeronautical 
Systems  Division,  Wright-Patterson  AFB,  Ohio;  and 
(3)  A-10  System,  which  is  being  procured  by  AFSC's 
Aeronautical  Systems  Division,  Wright-Patterson  AFB,  Ohio. 
The  three  major  weapon  system  programs  have  gone  through 
the  entire  PRR  process  beginning  with  the  planning  stage 
and  culminating  with  the  final  DSARC  III  review. 

Research  Hypothesis  No.  1 

Data  Collection 

The  interview  technique  was  used  as  the  means  of 
collecting  data  for  testing  Research  Hypothesis  No.  1. 
Personnel  in  two  types  of  government  offices,  the  major 
weapon  system  program*  office  (SPO)  and  the  Air  Force 
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Plant  Representative  Office  (AFPRO)  located  at  the 
pertinent  prime  contractor's  facility,  were  interviewed 
to  collect  data  on  each  of  the  three  major  weapon  system 
programs  which  have  completed  PRR  programs.  The  personnel 
within  the  SPOs  and  the  AFPROs  who  had  the  primary  respon- 
sibility for  conducting  the  PRR  programs  were  interviewed. 

Interview  Guide  No.  1,  which  is  furnished  in 
Appendix  A,  was  used  to  interview  the  personnel  in  both 
the  SPOs  and  the  AFPROs  for  all  three  major  system  programs. 
By  requesting  two  different  organizations  (i.e.,  SPOs  and 
AFPROs)  to  provide  data  on  standard  variables  for  each  PRR 
program,  internal  validity  of  data  gathering  was  enhanced. 

Identification/Definition 
of  Variables 

The  variables  analyzed  for  Research  Hypothesis 
No.  1 consisted  of  factors  that  could  be  objectively 
compared  to  determine  if  there  were  differences  of  prac- 
tical importance  among  the  three  AFSC  PRR  programs  that 
have  been  conducted  to  date.  The  variables  that  were 
measured  and  the  corresponding  measurement  scales  and  value 
levels  for  the  .variables  for  the  three  major  weapon  sys- 
tem programs  are  provided  in  Table  2 (pages  38-39) . Each 
variable  for  Research  Hypothesis  No.  1 is  identified  and 
operationally  defined  in  the  following  discussion. 

PRR  Visits  to  Prime  Contractor.  PRR  visits  to 
prime  contractor  are  the  total  number  of  visits  that  a 
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Table  2 — Continued 


39 


(0 

4) 

••  • -H 

- — - 03  J-i 
rH  4J  0 
<— ( <C  CP 
CM  TJ  4) 
••  P 
VO  4-1  (0 
'TOO 


in  o 

0)  0)  CM 
rH  -H 

(UPC 
>0(0 
<u  o>.c 

H 51  U 
4 J 


4)  (0  P <0 
3 O <U  P 
■— I P (0 
(0  Os  (0  T3 
> >H  <U 


I P 44 


Dsr-  C o 


C I I 


•H  I I 


S o>  <» 

0 IP  P 
rH  CXi’r4 

HV  C 

• 

0 p *P 

XX 

4-1  M V, 

p 

3 C 

c 

UlSlH  o 

xx  a 

p 

• 

• • 

p p 

XX  O P3 

CQ  (0 

P 

0)  rH 

•H 

P rH 

» • 

10  O 

(0 

0)  TJ 

0)  P • 

c 

P (0  (0 

p 

<U  T3  p 

O tn 

P (0 

P <u 

OPT) 

p 

(Q  O 

T3  «J 

•H  P 

<u  <u 

TJ  III  O 

P c 

<U 

p 

•0 

•h  m 

0 o 

<u 

P 0) 

ap 

•H 

O -H 

0) 

p 

O'  P 

P TJ 

• 

•H 

(U  0 

0) 

>i 

(0 

P O' 

E p . 

fH 

tn 

(0  a) 

(0  P >i  C 

(0 

O P 

P O -H 

0 

rH 

(0 

os  a c 

O CM  O 

0 <U  0 

» 

1 

p p 

(U 

<U 

1 VO 

a » 

•H 

P 

(0  1 

P <u 

> 

a 

3 co 

« U)  -H 

P 

O 1 

« o > 

<U 

(0  E 1 

a 

O P p 

a>  o 03 

<u  c 

rH  P 0) 

p 

e p -h 

XJ  O P 

0 

(0  c 

(0  rC  v> 

P -H  o 

•HOE 

c 

O'  ox 

P *P  'H 

0 

o o a 

•H 

p a a 

> 

p 

aw  < 

(0 

H • • 

p 

OX  P P 

»H  ITJ  *Q 

3 

ox  o 0 

< 

Q Cl,  Cm  Cm 

M 

<— 

®*  4- 

* H- 


_jrf 


I 


40 

system  program  office  PRR  team  of  any  size  made  to  review 
the  production  readiness  of  the  prime  contractor  using 
the  AFSCR  84-2  questions  as  guidance  (42:5-6). 

PRR  Visits  to  Subcontractors.  PRR  visits  to  sub- 
contractors are  the  total  number  of  visits  that  an  entire 
PRR  team  of  any  size  made  to  review  the  production  readi- 
ness of  any  subcontractor  using  AFSCR  84-2  as  guidance 
(42:5-6) . 

Different  Skills  (SPO) . Different  skills  (SPO) 
are  the  total  number  of  functional  skills  represented  in 
the  system  program  office’s  PRR  visits  to  any  contractor. 
Functional  skill  is  each  different  job  classification  as 
determined  by  an  individual's  position  description. 

PRR  Team  Members  (SPO) . PRR  team  members  (SPO) 
are  the  total  number  of  visits  to  any  contractor  made  by 
all  personnel  from  the  system  program  office  as  a partici- 
pant on  a PRR  team  using  AFSCR  84-2  as  a guide  (42:5-6). 
Each  visit  made  by  each  SPO  PRR  team  member  to  any  con- 
tractor was  counted  as  one  PRR  team  member  (SPO). 

Different  Skills  (AFPRO) . Different  skills  (AFPRO) 
are  the  total  number  of  functional  skills  represented 
from  the  local  Air  Force  Plant  Representative  Office 
(AFPRO)  at  all  PRR  meetings  held  with  the  prime  contractor 
or  any  subcontractors  using  AFSCR  84-2  as  guidance  (42:5-6). 
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Functional  skill  is  each  different  job  classif icatio:  as 
determined  by  an  individual's  position  description. 

PRR  Team  Members  (AFPRO) . PRR  team  members  (AFPRO) 
are  the  total  number  of  times  all  Air  Force  Plant  Repre- 
sentative Office  personnel  participated  as  a PRR  team 
member  on  any  review  held  at  the  prime  contractor's 
facility  or  any  visit  to  a subcontractor  using  AFSCR 
84-2  as  guidance  (42:5-6).  Each  different  visit  or 
participation  for  each  PRR  team  member  was  counted  as  one 
PRR  team  member  (AFPRO) . 

Supplemental  PRR  Team  Members . Supplemental  PRR 
team  members  are  the  total  number  of  visits  to  any  con- 
tractor made  by  all  personnel  not  in  the  system  program 
office  or  AFPRO  to  participate  on  a PRR  team  with 
AFSCR  84-2  as  guidance  (42:5-6).  Each  visit  made  by 
each  supplemental  PRR  team  member  to  any  contractor  was 
counted  as  one  supplemental  PRR  team  member. 

Duration  of  PRR  Program.  Duration  of  PRR  program 
is  the  total  amount  of  time  (in  months)  from  the  start  of 
the  planning  phase  of  the  PRR  program  up  to  the  time  of 
the  DSARC  III  review.  The  start  of  the  PRR  planning  phase 
is  defined  as  that  point  in  time  when  the  system  program 
office  actually  began  working  on  the  PRR  program. 
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Verbatim  AFSCR  84-2  PRR  Questions.  Verbatim 
AFSCR  84-2  PRR  questions  (Appendix  B)  are  the  total  number 
of  the  twenty-five  standard  AFSCR  84-2  PRR  questions 
(42:5-6)  that  were  used  on  any  PRR  visit  for  the  entire 
PRR  program.  Each  of  the  twenty-five  questions  was 
counted  only  once. 

Additional  PRR  Questions.  Additional  PRR  ques- 
tions are  the  total  number  of  questions  used  in  the  PRR 
program  that  were  either  modified  AFSCR  84-2  questions  or 
new  questions  addressing  other  areas  not  covered  by  the 
AFSCR  84-2  questions  (42:5-6).  Each  additional  PRR 
question  was  counted  only  once. 

PRR  Program  Cost  (SPO) . PRR  program  cost  (SPO) 
is  the  total  estimated  cost  incurred  by  the  system  pro- 
gram office  in  conducting  the  entire  PRR  program.  The 
estimate  included  PRR  contractual  coverage  expenses, 
temporary  duty  travel  expenses,  typing  expenses,  report 
publication  expenses,  manpower  costs,  etc.  that  could  be 
directly  attributed  to  the  conduct  of  the  entire  PRR 
program. 

PRR  Program  Cost  (AFPRO)  . PRR  program  cost  (AFPRO) 
is  the  total  estimated  cost  incurred  by  the  AFPRO  in 
conducting  the  entire  PRR  program.  The  estimate  includes 
temporary  duty  travel  expenses , typing  expenses , report 
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publication  expenses,  manpower  costs,  etc.  that  could  be 
directly  attributed  to  the  conduct  of  the  entire  PRR  pro- 
gram. 

Design  to  Test  Research 
Hypothesis  No.  1 

Since  a census  of  Population  I was  conducted  for 
Research  Hypothesis  No.  1,  statistical  inference  tech- 
niques were  not  appropriate  (46:185).  To  determine  the 
variability  in  the  PRR  approaches  to  date,  the  census 
data  were  tabulated  and  are  discussed  in  the  findings 
portion  of  the  research.  A coded  variability  (CV)  value 
was  computed  for  each  variable  for  each  of  the  three 
completed  PRR  programs.  The  basis  and  methodology  for 
determining  the  CV  values  are  explained  in  detail  in 
Appendix  C. 

Criteria  Test 

The  criteria  test  for  Research  Hypothesis  No.  1 
addressed  the  question  of  what  practical  importance  is 
a difference  in  the  PRR  approaches  taken  by  the  major 
weapon  system  programs.  The  criteria  test  for  Research 
Hypothesis  No.  1 was  determined  after  data  collection 
by  analyzing  the  actual  data  and  looking  for  large  average 
coded  variability  values  ( CV ) for  the  twelve  variables  among 
the  three  PRR  programs.  The  researchers  reasoned  that 
if,  on  the  average,  the  actual  data  values  exceeded  the 
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lowest  data  value  by  a factor  greater  than  one  for  a 
majority  of  the  twelve  variables,  then  Research  Hypothesis 
No.  1 would  be  supported.  An  expert  opinion  was  that  if 
eight  or  more  variables  in  Table  2 (page  38)  could  be 
determined  to  be  different  by  a CV  of  greater  than  1.5  for 
the  three  PRR  programs,  the  conclusion  would  then  be  that 
the  PRR  programs  were  conducted  differently  (20) . 

Research  Hypothesis  No.  2 

Data  Collection 

The  personal  interview  technique  was  also  the 
means  for  collecting  data  for  testing  Research  Hypothesis 
No.  2.  Interview  Guide  No.  2,  which  is  furnished  in 
Appendix  D,  was  used  to  conduct  the  interviews.  The 
survey  respondents  for  the  second  hypothesis  were  ten 
production  management  experts  within  Air  Force  Systems 
Command  (AFSC) . The  AFSC  production  management  experts 
were  required  to  possess  the  following  qualifications: 

(1)  active  in  the  production  management  area  for  at  least 
three  years,  (2)  involved  in  the  management  of  major  Air 
Force  weapon  systems  for  at  least  five  years,  and 
(3)  involved  in  the  planning  or  implementation  of  at 
least  one  AFSC  PRR  program.  The  key  determinant  for 
selecting  the  production  management  experts  was  their 
previous  PRR  experience.  The  selection  of  the  AFSC 
production  management  experts  was  limited  by  the  fact 
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that  few  AFSC  personnel  met  the  qualification  of  actually 
having  participated  in  a PRR  program  (20) . Therefore, 
the  selection  of  AFSC  production  management  experts  was 
judgmental.  The  AFSC  production  management  experts  who 
were  selected  for  the  study  effort  are  identified  in 

t 

Appendix  E. 

Identification/Definition 
of  Variables 

In  collecting  the  data  for  testing  Research 
Hypothesis  No.  2,  the  ten  AFSC  production  management 
experts  were  requested  to  rank  the  twenty-five  standard 
PRR  questions  listed  in  AFSCR  84-2  (42:5-6)  in  order  of 
importance  for  future  PRR  programs.  The  twenty-five 
AFSCR  84-2  PRR  questions  (42:5-6),  which  are  furnished 
in  Appendix  B,  are  the  variables  for  which  the  ordinal 
rankings  were  obtained.  Interview  Guide  No.  2 (Appendix  D) 
contains  the  following  specific  instructions  that  were 
used  to  request  the  experts  to  rank  the  PRR  questions: 

To  determine  if  a standard  PRR  approach  can  be 
developed  to  be  universally  and  practically  applied 
to  different  AFSC  weapon  system  programs,  rank  the 
twenty-five  standard  PRR  questions  in  AFSCR  84-*-2 
(42:5-6)  in  order  of  importance.  A "one"  should  be 
assigned  to  the  PRR  question  that  should  have  the 
greatest  amount  of  time  and  resources  expended  for 
analysis  and  reporting,  and  a "twenty  five"  should 
be  assigned  to  the  least  important  question . 

[Appendix  D,  page  122] . 
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Design  to  Test  Research 
Hypothesis  No.~~^ 

To  test  the  research  hypothesis  that  a standard  PRR 
approach  is  feasible,  the  following  nonparametric  Fried- 
man two-way  analysis-of-variance-by-ranks  test  (33:166) 
was  used.  The  null  hypothesis  (ff Q)  for  the  Friedman  test 
was  that  there  was  no  difference  among  the  mean  ranks  for 
the  twenty-five  AFSCR  84-2  PRR  questions;  therefore,  the 
ten  production  management  experts  would  not  agree  on  the 
rankings  for  the  twenty-five  questions.  The  alternate 
hypothesis  (H for  the  Friedman  test  was  that  there  were 
differences  among  the  mean  ranks  for  the  twenty-five  AFSCR 
84-2  PRR  questions;  therefore,  the  ten  experts  would  agree 
on  the  rankings  for  the  twenty-five  AFSCR  84-2  PRR  ques- 
tions. If  the  ten  AFSC  experts  agreed  on  all  twenty-five 
AFSCR  84-2  PRR  questions  (i.e.,  reject  H Q) , then  Research 
Hypothesis  No.  2 would  be  supported,  and  it  may  be  con- 
cluded that  application  of  the  AFSCR  84-2  questions  is 
universal  regardless  of  individual  weapon  system  program 
characteristics . 

Friedman  Two-Way  Analysis-of-Variance-by-Ranks  Test. 
Since  the  ranking  data  of  the  ten  AFSC  production  manage- 
ment experts  were  on  an  ordinal  scale,  the  Friedman  two-way 
analysis-of-variance-by-ranks  test  was  used  to  test  the 
null  hypothesis  that  the  different  columns  of  ranks  (sam- 
ples) from  the  ten  AFSC  production  management  experts  were 
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drawn  from  the  same  population  (33:166).  For  the  Friedman 
test,  it  was  assumed  that  the  observations  were  independent 
and  the  variables  under  study  had  underlying  continuity 
(33:31).  The  data  were  placed  in  a two-way  table  having 
N rows  and  k columns  (33:166).  The  rows  represented  the 
ten  AFSC  production  management  experts,  and  the  columns 
represented  the  twenty-five  AFSCR  84-2  PRR  questions 
(Table  3,  page  48).  Each  row  gives  the  rank  scores  of 
one  expert  for  the  twenty-five  AFSCR  84-2  PRR  questions. 

The  scores  in  each  row  were  ranked  independently  with  the 
ranks  in  any  one  row  ranging  from  one  through  twenty 
five.  The  Friedman  test  determined  if  the  columns  of  ranks 
came  from  the  same  population  (33:166). 

The  following  procedures  were  used  to  conduct  the 
Friedman  test  (33:166-172): 

1.  The  null  hypothesis  (# Q)  is  that  there  was  no 
difference  among  the  mean  ranks  for  the  twenty-five  AFSCR 
84-2  PRR  questions;  therefore,  the  ten  production  manage- 
ment experts  did  not  agree  on  the  rankings  for  the  twenty- 
five  AFSCR  84-2  PRR  questions  (42:5-6).  The  alternate 
hypothesis  (H is  that  there  were  differences  among  the 
mean  ranks  for  the  twenty-five  AFSCR  84-2  PRR  questions; 
therefore,  the  ten  experts  did  agree  on  the  rankings  for 
the  twenty-five  AFSCR  84-2  questions  (42:5-6). 

2.  The  level  of  significance  (alpha)  for  the 
Friedman  test  was  set  at  .05. 
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3.  The  rank  order  scores  were  placed  in  the  two- 
way  table  having  k columns  (one  column  for  each  of  the 
twenty-five  AFSCR  84-2  questions)  and  N rows  (one  row 
for  each  of  the  ten  AFSC  experts) . 

4.  The  sum  of  the  ranks  was  determined  for  each 

column  (J?  .)  . 

J 

2 

5.  The  value  of  Xranks  was  computed  in  the  fol- 
lowing manner: 

k 

Ws  ■ Nin^Z/V2-3N(k+1) 

j-1 

* Number  of  rows  in  two-way  table. 

* Number  of  columns  in  two-way  table. 

= Sum  of  ranks  in  jth  column. 

» Directs  one  to  sum  the  squares  of  the  sums 
of  ranks  over  all  k. 

6.  The  critical  value  for  the  .05  level  of  signif- 
icance test  was  determined  from  the  chi-square  distribution 

table  using  k-1  degrees  of  freedom. 

2 

7.  The  Xrankg  was  compared  against  the  critical 
value  obtained  in  Step  6.  If  the  X2rankg  exceeded  the 
critical  value,  the  conclusion  was  to  reject  H Q (the  null 
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hypothesis  of  no  difference  among  the  me"an  ranks)  . The 
conclusion  would  then  be  that  the  ten  AFSC  production 
management  experts  did  agree  on  the  rankings  for  the 
twenty-five  AFSCR  84-2  PRR  questions  (42:5-6). 

Friedman  Pilot  Study  Results 

Since  realistic  data  were  not  available  until  the 
actual  interviews,  a simulation  technique  was  used  to 
generate  ordinal  ranking  data  to  test  the  Friedman  two- 
way  analysis-of-variance-by-ranks  procedure  for  assessing 
agreement  among  the  ten  AFSC  production  management 
experts.  Random  numbers  from  "one"  to  "twenty  five" 
were  obtained  to  simulate  the  rankings  of  the  twenty- 
five  AFSCR  84-2  PRR  questions  for  each  of  the  ten  AFSC 
experts.  The  random  number  table  contained  in  the  CRC 
Standard  Mathematical  Tables  was  utilized  to  select  the 
ten  AFSC  experts'  simulated  responses  (32:628-632).  Ten 
different  starting  points  or  seeds  were  used,  and  dif- 
ferent progression  schemes  were  followed  to  select  the 
ordinal  rankings.  The  data  in  Table  4 (page  51)  were  col- 
lected by  following  the  simulation  procedure.  The  rows 
represented  the  ten  AFSC  production  management  experts, 
and  the  columns  represented  the  twenty-five  AFSCR  84-2 
PRR  questions.  Bach  row  represented  the  rank  scores  of 

one  expert  for  the  twenty-five  AFSCR  84-2  PRR  questions. 

2 

The  simulated  value  of  Xpan^8  was  computed  in  the 


following  manner: 


Table  4 — Continued 
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SOURCE:  U.S.  Air  Force  Systems  Command.  Production  Readiness  Review. 

AFSCR  84-2.  Washington,  D.C.,  23  November  1971,  ppT  5-6 . 
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X 


2 

ranks 


k 


j=l 


where : 

N = 10 
k = 25 

Rj  = provided  in  Table  4 (page  51)  for 

25  j=l32J...12S. 

^P(R.)2  = 437,505 

j=l 

So, 

Franks  * 116)  IKI  Hi)  («7 ,505) - (3)  (10)  (26) 

= 807.7015  - 780 

= 27.7015 

The  critical  value  for  the  .05  level  of  signifi- 
cance test  was  obtained  from  the  chi-square  distribution 

2 

taole  using  k-l=24  degrees  of  freedom.  So,  \ ...  y=36.4. 

QV'L  u "V  Q CL  l* 

2 

Since  7015  was  less  than  the  critical  value, 

the  conclusion  was  to  fail  to  reject  (the  null  hypoth- 
esis of  no  difference  among  the  mean  ranks) . 


By  using  artificial  data  from  the  random  number 
table,  there  was  insufficient  evidence  to  conclude  that 
the  ten  AFSC  production  management  experts  did  agree  on 
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the  rankings  for  the  twenty-five  AFSCR  84-2  PRR  questions. 
The  conclusion  was  that  the  ten  AFSC  production  experts 
did  not  agree  on  the  rankings  for  the  twenty-five  AFSCR 
84-2  PRR  questions.  Since  the  conclusion  in  the  simu- 
lated Friedman  test  was  to  fail  to  reject  it  indi- 

cated that  the  twenty-five  AFSCR  84-2  PRR  questions  could 
not  be  universally  applied  regardless  of  individual 
weapon  system  program  characteristics.  Since  the  artifi- 
cial data  were  randomly  generated,  and  since  the  random 
numbers  were  assigned  as  rank  order  values,  the  results 
of  the  Friedman  two-way  analysis-of-variance-by-ranks 
test  was  as  anticipated.  To  have  rejected  HQ  in  the 
Friedman  test,  the  random  numbers  that  were  generated 
for  the  test  would  have  had  to  simulate  actual  agreement 
among  the  ten  AFSC  experts'  rankings. 

Criteria  Test 

The  criteria  test  for  Research  Hypothesis  No.  2 
addresses  the  question  of  what  practical  importance  is 
standardization  in  the  selection  or  use  of  the  AFSCR  84-2 
PRR  questions  (42:5-6).  By  highlighting  the  AFSCR  84-2 
PRR  questions  that  should  receive  the  greatest  emphasis 
in  terms  of  resources  expended  regardless  of  program 
characteristics,  greater  returns  should  result  from  PRR 
program  inputs  (20) . If  the  Friedman  two-way  analysis- 
of-variance-by-ranks  test  (33:166-172)  determined  that  the 
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ten  AFSC  production  management  experts  did  agree  on  the 
rankings  for  all  twenty-five  AFSCR  84-2  PRR  questions 
(42:5-6)  at  the  .05  level  of  significance,  then  the  con- 
clusion would  be  that  a standard  PRR  approach  is  feasible. 

Generalization 

The  findings  in  support  of  Research  Hypothesis 
No.  1 were  not  generalized  outside  of  Population  I.  But, 
after  analyzing  the  data  for  Research  Hypothesis  No.  2, 
an  attempt  was  made  to  predict  the  factors  that  should 
be  emphasized  in  planning  and  implementing  PRR  programs 
for  future  Air  Force  major  weapon  systems. 

Validity  of  Measurement  Instruments 
The  measurement  instruments  that  were  used  for 
collecting  data  for  testing  Research  Hypotheses  No.  1 
and  No.  2 were  coordinated  with  production  management 
instructors  in  the  Air  Force  Institute  of  Technology 
Continuing  Education  Division  (13)  and  with  a Research 
Associate  in  the  Air  Force  Business  Research  Management 
Center  (20).  Comments  were  received  on  the  accuracy  and 
validity  of  the  measurement  instruments,  and  the  measure- 
ment instruments  were  modified  to  incorporate  recommended 
changes. 
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Summary  List  of  Assumptions 

1.  The  data  for  analyzing  Research  Hypothesis 
No.  1 were  independently  provided  by  individuals  from 
the  system  program  offices  and  the  Air  Force  Plant  Repre- 
sentative Offices. 

2.  Any  variables  omitted  in  data  collection  and 
analysis  had  no  significant  impact  on  the  research 
results . 

3.  All  estimates  supplied  by  the  data  sources 
reflect  the  real-world  situation. 

4.  Interview  respondents  interpreted  the  twenty- 
five  AFSCR  84-2  PRR  questions  (42:5-6)  in  the  same  man- 
ner. 

5.  For  the  nonparametric  statistical  tests,  the 
observations  are  independent,  and  the  variables  under  study 
have  underlying  continuity  (33:31). 

Summary  List  of  Limitations 

1.  Any  political,  economic,  technological,  and 
social  influences  beyond  the  control  of  the  major  weapon 
system  program  office  were  omitted  from  the  research. 

2.  Some  aspects  of  data  collection  and  analysis 
were  limited  by  the  researchers'  experience. 

3.  Some  of  the  original  PRR  program  planners  and 
participants  who  were  involved  in  the  three  completed  PRR 
programs  were  no  longer  available  for  interviews  in  the 


system  program  offices  or  in  the  Air  Force  Plant  Represen- 
tative Offices. 

4.  Some  variables  may  have  been  omitted  from  data 
collection  and  analysis. 

5.  Since  a limited  number  of  AFSC  production 
management  personnel  met  the  qualifications  established 
in  the  research,  a judgmental  selection  process  was  used 
to  obtain  the  AFSC  production  management  experts. 

6.  PRR  program  cost  data  were  not  tabulated  in 
the  format  requested,  so  the  best  estimates  of  the  data 


sources  were  used. 


CHAPTER  IV 


DATA  ANALYSIS 
Overview 

This  chapter  includes  the  specific  findings 
obtained  in  data  collection,  analyses  of  the  findings 
used  in  testing  Research  Hypotheses  No.  1 and  No.  2, 
and  corollary  findings  obtained  from  the  study  effort. 

Research  Hypothesis  No.  1 

Primary  Findings 

Census  data  were  collected  on  the  major  Production 
Readiness  Review  (PRR)  programs  which  have  been  conducted 
on  the  three  major  weapon  systems  that  have  undergone  the 
DSARC  III  decision-making  process — P-15,  A-10,  and  AWACS. 
Data  collected  on  each  PRR  program  included:  the  number  of 
PRR  visits  to  prime  contractor,  PRR  visits  to  subcontrac- 
tors, different  skills  (SPO) , PRR  team  members  (SPO) , PRR 
team  members  (AFPRO) , different  skills  (AFPRO) , supplemental 
PRR  team  members,  duration  of  PRR  program,  verbatim  AFSCR 
84-2  PRR  questions,  additional  PRR  questions,  estimated  PRR 
program  cost  (SPO) , and  estimated  PRR  program  cost  (AFPRO) . 
In  an  effort  to  analyze  Research  Hypothesis  No.  1,  ratio 
data  were  collected  on  the  tvalve  variables  listed  above. 

The  data  were  used  to  objectively  assess  if  differences 
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existed  in  the  conduct  of  the  three  major  PRR  programs 
completed  to  date.  The  data  collected  for  testing 
Research  Hypothesis  No.  1 are  provided  in  Appendix  F. 

The  following  significant  PRR  program  variations 
are  easily  recognizable  from  the  data  shown  in  Appen- 
dix F: 

1.  The  AWACS  program  office  made  three  major 
visits  to  the  prime  contractor  to  conduct  its  PRR  pro- 
gram, whereas  the  F-15  and  A-10  program  offices  made 
over  twenty  visits  to  their  respective  prime  contractors 
solely  to  conduct  PRR  program  activities.  However,  the 
F-15  PRR  reviewed  more  than  one  prime  contractor  (20) . 

2.  The  F-15  program  office  made  approximately 
fifty  visits  to  subcontractors  to  conduct  PRR  program 
activities,  while  the  A-10  and  AWACS  program  offices 
held  considerably  fewer  subcontractor  PRR  meetings. 

3.  There  were  fewer  system  program  office  (SPO) 
personnel  involved  in  the  AWACS  PRR  program  effort  than 
in  both  the  F-15  and  A-10  PRR  programs. 

4.  The  A-10  program  office  had  more  AFPRO  PRR 
team  members  than  did  both  the  F-15  and  AWACS  program 
offices.  This  could  indicate  more  AFPRO  involvement 
in  the  A-10  PRR  program  effort  than  in  the  other  two 
major  weapon  system  PRR  programs. 

5.  The  F-15  program  office  did  not  use  any  sup- 
plemental PRR  team  members,  whereas  the  A-10  and  AWACS 


program  offices  relied  heavily  on  organizations  other 
than  the  SPO  and  AFPRO  resources  for  additional  PRR 
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program  support. 

6.  The  F-15  PRR  program  was  conducted  in  five 
months,  whereas  the  A-10  and  AWACS  PRR  programs  lasted 
thirty  and  twenty-nine  months  respectively. 

7.  The  A-10  program  office  did  not  use  any  of 
the  AFSCR  84-2  PRR  questions  verbatim,  whereas  the  F-15 
and  AWACS  program  offices  used  most  of  the  AFSCR  84-2 
PRR  questions  verbatim. 

8.  The  A-10  program  office  used  approximately 
250  additional  PRR  questions,  while  the  F-15  and  AWACS 
program  offices  used  fewer  additional  evaluation  ques- 
tions in  their  respective  PRR  programs. 

9.  The  A-10  program  office  spent  considerably 
more  dollars  to  conduct  the  A-10  PRR  program  than  did 
both  the  F-15  and  AWACS  program  offices. 

To  objectively  test  Research  Hypothesis  No.  1, 
the  data  provided  in  Appendix  F were  coded  for  compari- 
son against  the  established  criteria  test.  The  coding 
methodology  is  provided  in  Appendix  C.  The  actual  calcu- 
lated average  coded  variability  values  (CV)  for  the 
twelve  variables  are  provided  in  Appendix  G. 


Summary  of  Primary  Findings 


The  easily  recognizable  differences  mentioned  above 


suggest  that  the  three  major  PRR  programs  were  conducted 
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differently.  The  same  indications  were  obtained  from  the 
coded  variability  calculations  provided  in  Appendix  G. 

Of  the  twelve  variables  used  for  testing  Research  Hypoth- 
esis No.  1,  nine  of  the  variables  had  average  coded  vari- 
ability values  ( CV ) in  excess  of  1.5.  The  interpretation 
of  these  results  is  that  when  comparing  each  of  the  three 
observations  for  each  variable  with  the  lowest  value  for 
that  particular  variable,  the  average  variability  for 
nine  of  the  twelve  variables  (75  percent)  used  for  test- 
ing Research  Hypothesis  No.  1 exceeded  the  established 
criteria  test  value  for  the  test,  which  had  been  set  at 
eight  of  the  twelve  total  variables. 

Research  Hypothesis  No.  2 

Primary  Findings 

To  evaluate  Research  Hypothesis  No.  2,  ten  Air 
Force  Systems  Command  (AFSC)  production  management  experts 
were  requested  to  prioritize  the  twenty-five  AFSCR  84-2 
PRR  questions.  If  the  ten  AFSC  experts  agreed  on  the 
relative  importance  of  evaluating  the  production  manage- 
ment areas  covered  in  the  twenty-five  AFSCR  84-2  PRR 
questions,  then  this  would  provide  support  for  the  hypoth- 
esis that  a standard  approach  is  feasible  for  the  conduct 
of  future  major  weapon  system  Production  Readiness 
Review  (PRR)  programs.  The  ten  AFSC  production  manage- 
ment experts  were  requested  to  independently  rank  order 
the  twenty-five  AFSCR  84-2  PRR  questions.  Each  expert 
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assigned  a "one"  to  the  PRR  question  that  was  considered 
most  important  and  a "twenty  five"  to  the  AFSCR  84-2  PRR 
question  of  least  importance. 

First  Primary  Finding 

The  actual  ranking  data  from  the  ten  AFSC  produc- 
tion management  experts  for  the  twenty-five  AFSCR  84-2  PRR 
questions  are  provided  in  Appendix  H.  The  primary  nonpara- 
metric  statistical  test  used  for  evaluating  the  agreement 
among  the  ten  AFSC  experts  for  the  twenty-five  AFSCR  84-2 
PRR  questions  was  the  Friedman  two-way  analysis-of- 
variance-by-ranks  test.  To  provide  additional  support 
for  the  results  obtained  from  the  Friedman  test,  two 
additional  statistical  tests  were  performed.  The  second 
test  was  the  nonparametric  Kendall  Coefficient  of  Con- 
cordance U,  and  the  third  statistical  test  used  for 
verification  purposes  was  the  parametric  F test  using  the 
Air  Force  Institute  of  Technology's  (AFIT)  Omnitab  II 
computer  program  (29:1-44). 

Friedman  Two-Way  Analysis-of-Variance-by-Ranks 
Test.  To  conduct  the  Friedman  two-way  analysis-of- 
variance-by-ranks  test,  the  data  were  placed  in  a two- 
way  table  having  ten  rows  and  twenty-five  columns.  The 
rows  represented  the  ten  AFSC  production  management 
experts,  and  the  columns  represented  the  twenty-five 
AFSCR  84-2  PRR  questions.  Each  row  gave  the  rank  scores  of 
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one  expert  for  the  twenty-five  AFSCR  84-2  PRR  questions. 
The  scores  in  each  row  were  ranked  independently  with  the 
ranks  in  any  one  row  ranging  from  one  through  twenty  five. 
The  Friedman  test  determined  if  the  columns  of  ranks  came 
from  the  same  population  and,  hence,  whether  the  ten 
experts  agreed  on  the  relative  importance  of  the  twenty- 
five  AFSCR  84-2  PRR  questions. 

The  specific  procedure  used  in  conducting  the 
Friedman  test  was  covered  in  the  methodology  section  of 
this  study,  but  the  null  hypothesis  (H Q)  and  the  alternate 
hypothesis  are  iterated  at  this  time  for  the  reader's 

convenience.  The  null  hypothesis  (H Q)  was  that  there 
would  be  no  difference  among  the  mean  ranks  for  the  twenty- 
five  AFSCR  84-2  PRR  questions;  therefore,  the  ten  AFSC 
production  management  experts  would  not  have  agreed  on  the 
rankings  for  the  twenty-five  AFSCR  84-2  PRR  questions. 

The  alternate  hypothesis  (H was  that  there  would  be  a 
difference  among  the  mean  ranks  for  the  twenty-five 
AFSCR  84-2  PRR  questions;  therefore,  the  ten  AFSC 
production  management  experts  would  have  agreed  on  the 
rankings  for  the  twenty-five  AFSCR  84-2  PRR  questions. 


The  actual  value  of  X^anJig  was  computed  as 


follows : 


kranks 


Nk 


vhrr%2(*)>2~M(k*1) 


j-i 


where : 

N » Number  of  rows  in  two-way  table, 
k ■ Number  of  columns  in  two-way  table. 

Rj  - Sum  of  ranks  in  jth  column, 
k 

l ■ Directs  one  to  sum  the  squares  of  the  sums 
j«l 

of  ranks  over  all  k. 
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So, 


Ws  - [oomfiw]'455-1101  - <3H10U26) 

= 840.2031  - 780.0000 
- 60.2031 

The  critical  value  for  the  .05  level  of  signifi- 
cance test  was  determined  from  the  chi-square  distribution 
table  using  k-l*25-l~24  degrees  of  freedom.  The  critical 

value  for  .05  level  of  significance  and  24  degrees  of  freedom 

2 

was  determined  to  be  36.4.  Since  the  y . * 60.2031  exceeded 

panK.8 

the  critical  value  of  36.4,  the  conclusion  was  made  to 
reject  UQ  (the  null  hypothesis  of  no  difference  between  the 
mean  ranks) . The  conclusion  was  that  the  ten  AFSC  produc- 
tion management  experts  did  agree  on  the  rankings  for  the 
twenty-five  AFSCR  84-2  PRR  questions.  Since  the  conclusion 
was  to  reject  HQ,  it  may  be  possible  to  apply  the  AFSCR  84-2 
PRR  questions  universally  regardless  of  individual  weapon 
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system  program  characteristics.  The  Friedman  nonparametric 
test  is  conservative  in  nature  (i.e.,  conservative  with 
respect  to  rejection  of  H Q) . Nonparametric  tests  are 
appropriate  in  those  cases  in  which  information  concerning 
the  distribution  properties  of  population  parameters  is 
limited  or  not  obtainable.  As  a result  of  such  limita- 
tions, the  conclusions  reached  in  analyzing  these  data 
are  more  general  in  nature  and  the  test  of  H Q less 
powerful  than  if  parametric  methods  requiring  stronger 
assumptions  had  been  utilized  (33:19-21). 

To  further  test  the  null  hypothesis  (H Q)  in  the 
Friedman  test,  two  additional  level-of-signif icance 
values  were  used  to  determine  if  the  initial  conclusion 
would  be  changed.  The  critical  value  at  the  .01  level 
of  significance  with  24  degrees  of  freedom  was  determined 
to  be  43.0,  and  the  critical  value  at  the  .005  level  of 
significance  with  24  degrees  of  freedom  was  found  to  be 

45.6.  In  both  cases,  the  conclusion  was  the  same  as 

2 

before.  The  y i * 60.2031  exceeded  the  critical  values, 

A ranks 

so  Hq  was  rejected  in  both  cases.  The  final  conclusion 
from  the  Friedman  test  was  that  the  ten  AFSC  production 
management  experts  did  agree  on  the  rankings  for  the 
twenty-five  AFSCR  84-2  PRR  questions. 

To  verify  the  results  obtained  from  the  nonpara- 
metric Friedman  two-way  analysis-of-variance-by-ranks 
test,  two  additional  statistical  tests  were  conducted — 


the  nonparametric  Kendall  Coefficient  of  Concordance  W 
and  the  parametric  F test  generated  from  an  Omnitab  II 
computer  program  (29:1-44). 

Kendall  Coefficient  of  Concordance  W.  The  Kendall 
Coefficient  of  Concordance  W (33:239)  is  useful  for 
determining  the  agreement  among  several  experts  or  the 
association  among  three  or  more  variables.  The  Kendall 
Coefficient  of  Concordance  W has  special  applications 
in  providing  a standard  method  of  ordering  entities 
(i.e.,  AFSCR  84-2  PRR  questions)  according  to  consensus 
when  there  is  available  no  objective  order  of  entities 
(33:239).  The  actual  procedure  followed  in  conducting 
the  nonparametric  Kendall  Coefficient  of  Concordance  W 
test  is  provided  in  Appendix  I.  The  null  hypothesis 
(H Q)  for  the  Kendall  Coefficient  of  Concordance  V is 
that  the  experts'  rankings  are  unrelated  to  each  other. 

The  observations  provided  in  Appendix  H were  used  to 
make  the  Kendall  Coefficient  of  Concordance  W calcula- 
tions. The  actual  value  of  the  mean  of  the  R . was  computed 


as  follows: 
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where : 

N = Number  of  entities  to  be  ranked  (25). 

Rj  = Sum  of  the  ranks  assigned  to  each  entity. 

17  = Mean  of  the  R . values . 

3 3 


The  sum  of  the  squared  deviations  (s)  of  all  twenty-five 

R . values  from  R . was  calculated  as  shown  below: 

3 3 


s 


Z (Rj -Rj ) 


2 


= 32,610 


An  actual  value  for  W was  computed  as  follows: 


W 


_s 

(N3-N) 


32,610 

i-(100)  (15,600) 


= .25084617 


where : 

k » Number  of  experts  assigning  ranks  (10) . 

Since  N was  larger  than  seven,  the  following  formula  was 

2 

used  to  calculate  a value  for  x • 
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x2  = § = k (N-l ) W 

^kN(N+l) 

= 10  (24)  (.25084617) 

= 60.20308 

Referring  to  the  table  of  critical  values  of  chi 

2 

square  (33:249),  it  was  found  that  the  x greater  than 
or  equal  to  60.20308  with  degrees  of  freedom  equal  to 
N- 1=24  has  a probability  of  occurrence  under  EQ  of  less 
than  .001.  It  can  be  concluded  with  considerable  assur- 
ance that  the  agreement  among  the  ten  AFSC  production 
management  experts  was  higher  than  it  would  be  by  chance. 
Since  the  probability  under  H associated  with  a W 
value  of  .25084617  was  very  low,  the  null  hypothesis 
(H Q)  that  the  experts'  rankings  are  not  related  to  each 
other  can  be  rejected.  The  Kendal 1 Coefficient  of  Con- 
cordance W results  gave  credence  to  the  outcome  obtained 
from  the  Friedman  two-way  analysis-of-variance-by-ranks 
test. 

Parametric  F Test.  To  lend  further  credence  to 
the  results  obtained  from  the  Friedman  two-way  analysis- 
of-variance-by-ranks  test  and  the  Kendall  Coefficient  of 
Concordance  W test,  the  parametric  F test  was  used  in 
conducting  a one-way  analysis  of  variance  on  the  rankings 
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obtained  from  the  ten  AFSC  production  management  experts 
provided  in  Appendix  H.  The  F test, while  parametric, was 
not  applied  to  the  data  as  if  the  data  permitted  the 
assumptions  required  for  the  F test.  However,  when  the 
results  of  the  F test  are  identical  to  nonparametric 
test  results,  the  nonparametric  results  are  usually  further 
supported  (17)  . A computer  library  program  (Omnitab  II) 
was  used  to  calculate  the  values  associated  with  a one- 
way analysis  of  variance  (29:1-44).  The  ten  by  twenty- 
five  matrix  of  values  was  entered  into  the  time  sharing 
remote  terminals  of  the  CREATE  computer  at  the  Air  Force 
Institute  of  Technology  School  of  Systems  and  Logistics. 

The  output  obtained  from  the  Omnitab  II  computer  package 
is  provided  in  Appendix  J. 

The  calculated  F ratio  value  furnished  in  Appen- 
dix J is  3.139.  The  F critical  value  for  a .05  level  of 
significance  with  24  upper  degrees  of  freedom  and  225 
lower  degrees  of  freedom  is  1.52  (32:622).  Since  the 
calculated  F value  of  3.139  is  greater  than  the  critical 
F value  of  1.52,  the  conclusion  was  made  to  reject  HQ 
(the  null  hypothesis  of  no  difference  between  the  experts' 
rankings) . There  was  significant  evidence  to  conclude 
that  the  ten  AFSC  production  management  experts  did  agree 
on  the  relative  importance  of  applying  the  twenty-five 
AFSCR  84-2  PRR  questions  in  a PRR  program.  So,  both 
nonparametric  statistical  tests — the  Friedman  two-way 
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analysis-of-variance-by-ranks  test  and  the  Kendall  Coef- 
ficient of  Concordance  W — and  the  parametric  F 
test  in  the  one-way  analysis  of  variance  evidenced 
the  same  conclusion  that  the  ten  AFSC  production  manage- 
ment experts  did  agree  on  the  application  and  importance 
of  the  twenty-five  AFSCR  84-2  PRR  questions. 


AFSCR  84-2  PRR  Question  Priority  List.  Since 
the  ten  AFSC  production  management  experts  did  agree  on 
the  relative  importance  of  all  twenty-five  AFSCR  84-2 
PRR  questions,  the  medians  and  means  of  the  ten  experts' 
rankings  were  determined  for  each  one  of  the  twenty-five 
AFSCR  84-2  PRR  questions.  The  median  and  mean  calcula- 
tions are  included  in  Appendix  H.  The  PRR  question  that 
had  the  lowest  mean  or  median  value  for  the  ten  rankings 
was  identified  as  the  PRR  question  that  should  receive 
the  most  emphasis  in  a future  PRR  program,  and  the  PRR 
question  with  the  highest  mean  or  median  for  the  ten 
rankings  should  receive  the  least  emphasis  in  future  PRR 


programs . 

A comparison  of  the  rankings  of  the  twenty-five 
AFSCR  84-2  PRR  questions  using  the  medians  and  the  means 
for  each  column  of  experts'  rankings  is  provided  in 
Appendix  K.  As  shown  in  Appendix  K,  the  relative  rankings 
using  either  the  medians  or  means  are  very  similar.  But, 
the  researchers  chose  to  use  the  means  of  the  ten  rankings 
for  each  question  for  two  primary  reasons  vl7) . First, 
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there  are  fewer  ties  in  the  list  of  rankings  for  the 
twenty-five  AFSCR  84-2  PRR  questions,  so  the  researchers 
reduced  the  subjectivity  in  placement  of  some  of  the 
tied  rankings.  Secondly,  by  using  the  means,  the 
researchers  were  using  the  more  parametrically  efficient 
of  the  two  central  tendency  measures  (51:151).  The 
means  take  into  consideration  the  extreme  values  in  the 
data  set.  Using  the  mean  values  for  the  ten  rankings 
for  each  of  the  twenty-five  AFSCR  84-2  PRR  questions,  the 
researchers  developed  a priority  list  of  PRR  questions 
from  most  important  to  least  important  (Appendix  L) . All 
ties  are  so  indicated  in  Appendix  L.  Since  there  were 
■ only  two  pairs  of  tied  rankings,  the  researchers  determined 

that  the  tied  rankings  would  have  no  significant  impact 
on  the  research  results,  regardless  of  the  order  of 
placement  in  the  prioritized  list  of  AFSCR  84-2  PRR  ques- 
tions. For  each  pair  of  tied  rankings,  the  smaller 
numbered  question,  as  listed  in  the  present  AFSCR  84-2, 
is  provided  first  in  the  prioritized  list  of  PRR  questions 
(Appendix  L) . 

Second  Primary  Finding 

In  addition  to  the  fact  that  the  ten  AFSC  pro- 
duction management  experts  agreed  on  the  relative  impor- 
tance of  all  twenty-five  AFSCR  84-2  PRR  questions,  the 
following  other  primary  findings  were  obtained  from  the 
data  collection  activity  for  Research  Hypothesis  No.  2. 


4 
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The  second  primary  finding  was  that  six  of  the  twenty-five 

AFSCR  84-2  PRR  questions  had  mean  Rj  values  much  lower  than 

the  other  nineteen  mean  R.  values.  These  six  PRR  questions 

J 

were  AFSCR  84-2  PRR  question  numbers  1 (producible  engi- 
neering design) , 2 (development  engineering  problems) , 

5 (design  changes),  6 (advanced  production  planning), 

13  (production  impact  of  unresolved  technical  problems) , 
and  14  (test  program  results) . A review  of  the  six  AFSCR 
84-2  PRR  questions  above  indicated  that  five  of  the  six 
questions  addressed  the  completion  of  engineering  and 
testing  in  some  manner. 

Third  Primary  Finding 

In  the  interviews  conducted  to  collect  data  to 
test  Research  Hypothesis  No.  1,  there  was  a general 
consensus  of  opinion  expressed  by  the  respondents  that 
the  twenty-five  AFSCR  84-2  PRR  questions  were  for  the 
most  part  comprehensive,  covering  most  of  the  relevant 
areas  for  conducting  a PRR  program  for  practically  any 
type  of  weapon  system  acquisition.  The  consensus  ranking 
of  the  twenty-five  AFSCR  84-2  PRR  questions  by  the  ten 
AFSC  production  management  experts  (Appendix  L)  did  not 
agree  with  the  order  of  the  AFSCR  84-2  PRR  questions 
provided  in  the  present  AFSCR  84-2  document.  Interpreta- 
tion of  this  finding  is  that  the  present  regulation  does 
not  prioritize  the  twenty-five  AFSCR  84-2  PRR  questions  in 
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a manner  useful  to  the  system  program  office  charged  with 
the  responsibility  of  conducting  a Production  Readiness 
Review  (PRR)  program. 

Summary  of  Primary  Findings 

The  findings  for  Research  Hypothesis  No.  2 center 
around  the  fact  that  the  ten  AFSC  production  management 
experts  did  agree  on  the  relative  importance  and  applica- 
tion of  the  twenty-five  AFSCR  84-2  PRR  questions  for  future 
major  weapon  system  programs.  A group  of  engineering  and 
testing  completion  related  questions  was  placed  at  the  top 
of  the  list  of  prioritized  AFSCR  84-2  PRR  questions,  and 
there  was  considerable  agreement  on  the  high  importance 
of  assuring  that  the  engineering  tasks  were  completed 
prior  to  the  start  of  the  production  phase  of  the  weapons 
acquisition  process.  In  addition,  the  current  AFSCR  84-2 
provides  a list  of  twenty-five  AFSCR  84-2  PRR  questions 
that  does  not  agree  with  the  prioritized  list  of  the  same 
twenty-five  questions  from  the  ten  AFSC  production  manage- 
ment experts  that  have  had  previous  PRR  program  experience. 

Corollary  Findings 

This  research  effort  was  directed  to  the  collection, 
statistical  analysis,  and  interpretation  of  data  to  deter- 
mine first  the  differences  among  the  three  major  PRR 
programs  that  have  been  completed  to  date  and  secondly 
the  feasibility  of  using  a standard  approach  to  accomplish 
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the  Production  Readiness  Review  (PRR)  task.  In  the  pre- 
ceding sections  of  this  chapter,  the  primary  findings  were 
presented.  Corollary  findings  provided  in  this  section 
are  based  on  collected  data,  review  of  Air  Force  publi- 
cations pertaining  to  PRR,  discussions  with  production  and 
procurement  personnel  who  have  been  actively  involved  with 
PRR  activity,  and  the  researchers'  interpretations  of 
their  studies  and  observations.  Support  for  the  two 
research  hypotheses  depends  on  the  primary  findings. 

However,  the  researchers  believe  that  the  corollary  find- 
ings provide  additional  support  for  the  research  conclu- 
sions . 

First  Corollary  Finding 

An  attempt  was  made  in  this  study  to  conduct  an 
initial  survey  concerning  the  following  question:  When 

should  the  PRR  program  activity  actually  begin  in  the 
weapons  acquisition  process?  The  ten  AFSC  production 
management  experts  were  requested  to  indicate  prior  to 
which  DSARC's  (i.e.,  DSARC  I,  II,  and/or  III)  should 
the  twenty-five  AFSCR  84-2  PRR  questions  be  evaluated. 

The  ten  AFSC  production  management  experts  were  told  that 
any  combination  of  responses  could  be  provided  for  this 
initial  survey.  The  specific  responses  from  the  ten  AFSC 
experts  are  provided  in  Table  5 (page  75) . As  shown 
in  Table  5 (page  75),  all  ten  AFSC  experts  agreed  that  all 
twenty-five  AFSCR  84-2  PRR  questions  should  be  evaluated 
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*Pive  or  more  of  the  ten  AFSC  production  management  experts  agreed  that 
these  twenty  AFSCR  84-2  PRR  questions  should  be  evaluated  prior  to  DSARC  II. 


76 


prior  to  DSARC  III.  In  addition,  all  ten  AFSC  production 
management  experts  agreed  that  AFSCR  84-2  PRR  question 
numbers  seven  (standardization  of  design)  and  eleven 
(make-or-buy  structure)  should  be  evaluated  prior  to 
DSARC  II.  Nine  of  the  ten  AFSC  experts  agreed  that  AFSCR 
84-2  question  numbers  one  (producible  engineering  design) 
and  twenty  (laboratory  or  model  shop  constraints)  should 
be  evaluated  prior  to  DSARC  II.  In  summary,  five  or 
more  of  the  ten  AFSC  production  management  experts  agreed 
that  twenty  of  the  twenty-five  AFSCR  84-2  PRR  questions 
should  be  evaluated  prior  to  DSARC  II.  Some  of  the 
experts  contended  that  some  of  the  AFSCR  84-2  PRR  ques- 
tions should  be  initially  evaluated  prior  to  DSARC  I 
(refer  to  Table  5,  page  75).  The  consensus  of  the  ten 
AFSC  production  management  experts  was  that  initial  PRR 
activities  should  be  conducted  much  earlier  than  just 
prior  to  the  preparations  for  the  DSARC  III  decision- 
making process. 

Second  Corollary  Finding 

It  was  the  opinion  of  most  of  the  interview  respon- 
dents that  the  present  AFSCR  84-2  document  does  not  provide 
sufficient  guidance  on  how  to  conduct  a PRR  program. 

The  current  AFSCR  84-2  document  is  vague  in  some  key 
areas  and  incomplete  in  others.  For  example,  there  is 
a general  inference  in  AFSCR  84-2  that  a PRR  should  be 
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a one-time  effort  accomplished  just  prior  to  DSARC  III. 

The  weakness  of  this  inference  was  highlighted  by  the 
interview  respondents  and  by  the  results  provided  in  the 
first  corollary  finding.  Another  major  weakness  of 
the  present  AFSCR  84-2  highlighted  by  the  interview 
respondents  was  the  lack  of  sufficient  guidance  on  the 
PRR  team  composition.  The  present  AFSCR  84-2  primarily 
suggests  production  management  personnel  as  team  members 
and  does  not  emphasize  the  fact  that  design  engineers 
and  other  procurement  personnel  should  be  included  on  the 
PRR  teams  as  required.  The  respondents  also  indicated  that 
the  present  AFSCR  84-2  provides  insufficient  guidance  on 
the  extent  of  contractual  coverage  necessary  to  conduct 
a PRR  program.  PRR  programs  are  presently  being  con- 
ducted under  different  levels  of  contractual  coverage 
ranging  from  full  contract  coverage  to  little  or  no 
contract  coverage  for  PRR  activities. 

Third  Corollary  Finding 

The  general  consensus  of  the  personnel  interviewed 
was  that  there  is  a lack  of  readily  available  documenta- 
tion on  how  previous  PRR  programs  have  been  conducted 
to  date.  Each  major  weapon  system  PRR  program  plan 
prepared  to  date  was  developed  from  the  beginning.  No 
readily  accessible  lessons-learned  information  is  avail- 
able for  those  programs  that  are  now  required  by  AFSCR 
84-2  to  conduct  PRR  programs.  In  addition,  the  consensus 
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of  the  interviewees  was  that  there  was  no  Air  Force  expert 
available  in  a staff  level  position  who  could  provide 
assistance  to  a system  program  office  to  help  develop  the 
initial  PRR  program  plan.  Historically,  it  has  been 
incumbent  on  each  system  program  office  to  determine  what 
previous  PRR  programs  had  been  conducted,  contact  personnel 
that  had  been  involved  in  those  previous  PRR  programs  to 
determine  what  had  actually  been  accomplished  in  each  PRR 
program,  and  finally  develop  a new  PRR  program  plan,  using 
their  research  information  as  a baseline,  that  would  best 
fit  their  particular  weapon  system  program. 

Fourth  Corollary  Finding 

In  collecting  data  for  testing  Research  Hypothesis 
No.  1,  it  became  evident  that  accurate  cost  data  on  what 
it  actually  did  cost  the  Air  Force  to  conduct  PRR  programs 
to  date  are  lacking.  As  a result,  the  PRR  program  cost 
figures  supplied  by  the  system  program  offices  and  the 
Air  Force  Plant  Representative  Offices  for  testing 
Research  Hypothesis  No.  1 were  very  rough  estimates  on 
what  the  PRR  program  cost  each  organization.  In  addition, 
no  data  were  available  on  the  potential  savings  that  may 
have  accrued  to  the  Air  Force  as  a result  of  conducting 
PRR  programs.  The  researchers  were  unable  to  obtain 
any  documented  cost  or  savings  data  for  the  three  major 
Air  Force  PRR  programs  that  have  been  completed  to  date. 
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Fifth  Corollary  Finding 

AFSCR  84-2  is  directed  at  the  major  weapon  system 
programs  as  defined  by  DOD  Directive  5000.1  as  having 
$50  million  for  development  or  $200  million  for  produc- 
tion (42:1).  The  AFSCR  84-2  does  not  provide  much  assis- 
tance for  the  smaller  Air  Force  Systems  Command  programs 
that  are  just  under  the  threshold  levels.  But,  system 
program  office  personnel  for  the  smaller  programs  are 
still  tasked  to  make  an  assessment  about  the  contractor's 
readiness  to  transition  from  the  full-scale  development 
phase  into  the  production  phase  of  the  weapons  acquisition 
process.  Several  interview  respondents  expressed  the 
opinion  that  a mini-PRR  program  should  be  conducted  on 
the  smaller  Air  Force  Systems  Command  weapon  system 
programs . 

Sixth  Corollary  Finding 

The  information  gleaned  from  the  background  review 
indicated  that  the  Air  Force  is  the  only  military  depart- 
ment that  conducts  a PRR  type  of  analysis  prior  to  move- 
ment into  the  production  phase  of  the  weapons  acquisition 
process.  The  other  military  departments  conduct  con- 
tinuous surveys  and  analyses  in  this  area,  but  no  formal- 
ized procedure  or  system  has  been  developed  to  accomplish 
the  task.  Additional  interviews  conducted  after  the 
initial  background  review  produced  similar  results.  Based 
on  our  research  findings,  the  Air  Force  remains  as  the 
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only  military  department  that  has  a formal  PRR  procedure. 
Since  movement  into  the  production  phase  of  the  weapons 
acquisition  process  is  very  critical  and  expensive,  the 
researchers  conclude  that  the  other  military  departments 
should  consider  establishing  similar  procedures  to 
ensure  that  a contractor  is  ready  to  make  the  transition 
from  the  full-scale  development  phase  into  the  production 
phase  of  the  weapons  acquisition  process. 


CHAPTER  V 


CONCLUSIONS 

Overview 

This  chapter  contains  conclusions  related  to  the 
findings  that  were  discussed  in  the  previous  chapter. 

This  research  effort  had  two  major  objectives.  The  first 
objective  was  to  determine  what  disparities  existed  among 
the  three  PRR  programs  which  have  been  conducted  on  major 
Air  Force  weapon  system  programs  to  date.  The  second 
major  objective  was  to  determine  if  a standard  PRR  approach 
is  feasible  in  determining  whether  a contractor  is  ready 
to  transition  from  the  full-scale  development  phase  into 
the  production  phase  of  the  weapons  acquisition  process. 

The  primary  conclusions  are  followed  by  corollary 
conclusions.  The  corollary  conclusions  are  provided  so 
that  a more  complete  picture  of  the  Production  Readiness 
Review  (PRR)  process  will  be  presented.  The  corollary 
conclusions  were  formed  as  interviews  were  conducted, 
documents  were  reviewed,  pertinent  data  were  gathered, 
and  from  personal  observations  of  organizational  and 
personal  interactions  in  the  organizations  responsible 
for  the  PRR  programs. 
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Research  Hypothesis  No.  1 
Air  Force  major  weapon  system  programs  utilize 
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different  Production  Readiness  Review  (PRR)  program 
approaches  to  comply  with  AFSCR  84-2. 


ft. 


Hypothesis  Support 

Research  Hypothesis  No.  1 was  supported.  A visual 
inspection  of  the  twelve  variables  in  Appendix  F indi- 
cated that  some  factors  in  each  program  were  substantially 
different  from  the  same  factors  in  other  programs.  In 
addition,  the  average  coded  variability  values  for  the 
twelve  variables  used  in  testing  Research  Hypothesis  No.  1 
provided  similar  results.  Nine  of  the  twelve  average 
coded  variability  values  were  over  1.5,  which  indicated 
a substantial  average  variability  among  the  three  PRR 
programs  for  the  twelve  variables.  The  established  cri- 
teria test  of  eight  of  the  twelve  variables  for  Research 
Hypothesis  No.  1 was  exceeded. 


General  Conclusions 

The  primary  conclusion  after  testing  Research 
Hypothesis  No.  1 was  that  the  PRR  approaches  used  to 
date  have  been  different.  So,  the  first  research  objec 
tive  was  accomplished,  since  it  was  discovered  that 
significant  disparities  did  exist  among  the  three  PRR 
program  approaches  that  have  been  completed  to  date. 

The  specific  reasons  for  using  the  different  approaches 
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could  not  be  determined.  But,  the  conclusion  is  that  it 
should  not  be  necessary  to  start  at  the  beginning  for 
every  aspect  of  a new  PRR  program.  At  least  some  factors 
or  activities  should  remain  essentially  the  same  for  future 
PRR  programs.  To  test  this  contention.  Research  Hypothe- 
sis No.  2 was  developed. 

Research  Hypothesis  No.  2 

Air  Force  Systems  Command  major  weapon  system 
program  offices  can  follow  a standard  Production  Readi- 
ness Review  (PRR)  approach  to  comply  with  AFSCR  84-2. 

Hypothesis  Support 

The  researchers  contended  that  standardization 
could  be  obtained  in  conducting  future  PRR  programs  if 
there  were  agreement  among  the  AFSC  experts  who  have  had 
previous  PRR  experience  about  how  best  to  conduct  future 
PRR  programs.  So,  the  researchers  determined  that  one 
way  to  test  the  agreement  and  feasibility  of  using  a 
standard  PRR  approach  was  to  have  the  ten  AFSC  production 
management  experts  rank  order  the  twenty-five  questions 
that  are  provided  in  AFSCR  84-2  as  guidance  for  reviewing 
various  aspects  of  a weapon  system  program  to  determine 
the  program's  readiness  to  transition  into  the  produc- 
tion phase  of  the  weapons  acquisition  process.  The  data 
collection  and  analysis  indicated  that  the  ten  AFSC 
production  management  experts  did  agree  on  the  relative 
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importance  of  evaluating  the  twenty-five  AFSCR  84-2  PRR 
questions.  So,  this  conclusion  provided  support  for  the 
hypothesis  that  a standard  PRR  approach  is  feasible  for 
application  in  future  weapon  system  acquisition  programs. 

General  Conclusions 

This  section  addresses  the  general  . inclusions 
from  the  three  primary  findings  obtained  in  the  data 
analysis  for  Research  Hypothesis  No.  2.  As  shown  in  the 
first  primary  finding  for  Research  Hypothesis  No.  2,  a 
prioritized  list  should  indicate  to  AFSC  system  program 
offices  where  the  emphasis  should  be  placed  in  future  PRR 
program  planning  activities.  The  prioritized  list  is  a 
consensus  of  opinion  from  Air  Force  Systems  Command 
(AFSC)  experts  who  have  experienced  the  entire  PRR  process 
through  the  stages  of  planning,  implementation,  and 
final  DSARC  III  activity.  So,  a major  weapon  system 
program  office  should  be  able  to  take  the  prioritized  list 
of  AFSCR  84-2  PRR  questions  and  develop  a PRR  program 
around  the  prioritized  list.  The  major  emphasis  in 
future  PRR  programs  should  be  placed  in  those  areas 
addressed  by  the  AFSCR  84-2  PRR  questions  at  the  top  of 
the  prioritized  list,  and  lesser  emphasis  should  be 
placed  in  those  areas  addressed  by  the  lower  ranked 
AFSCR  84-2  PRR  questions. 
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As  shown  by  the  second  primary  finding,  there 
were  six  of  the  twenty-five  AFSCR  84-2  PRR  questions 
that  were  ranked  much  higher  in  importance  over  the 
remaining  nineteen  questions.  The  researchers  conclude 
that  the  results  on  these  six  questions — AFSCR  84-2  PRR 
question  numbers  1 (producible  engineering  design) , 

2 (development  engineering  problems) , 5 (design  changes) , 

6 (advanced  production  planning) , 13  (production  impact 
of  unresolved  technical  problems) , and  14  (test  program 
results) — indicate  that  a system  program  office  should 
place  greater  emphasis  on  these  six  questions  in  a PRR 
program.  It  was  also  concluded  that  a contractor  should 
be  tasked  to  respond  to  those  six  AFSCR  84-2  PRR  ques- 
tions in  greater  detail  and  earlier  in  the  system  pro- 
gram office's  preparations  for  the  DSARC  III  decision- 
making process.  In  addition,  since  five  of  the  top  six 
prioritized  AFSCR  84-2  PRR  questions  pertain  to  comple- 
tion of  engineering  and  testing,  the  system  program  office 
should  concentrate  on  assuring  that  the  majority  of  the 
engineering  activity  has  been  completed  and  that  the  con- 
tractor is  ready  to  begin  the  production  phase  of  the 
weapons  acquisition  process. 

The  third  primary  finding  indicated  that  the  cur- 
rent list  of  AFSCR  84-2  PRR  questions  is  fairly  compre- 
hensive, but  the  order  of  the  questions  in  AFSCR  84-2  in 
no  way  agrees  with  the  manner  in  which  the  ten  AFSC 
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production  management  experts  perceive  the  relative  impor- 
tance of  applying  the  twenty-five  AFSCR  84-2  questions  in 
a PRR  program.  The  present  format  of  the  twenty-five 
AFSCR  84-2  PRR  questions  tends  to  confuse  the  require- 
ments for  accomplishing  the  PRR  task.  The  researchers 
conclude  that  the  present  regulation  should  indicate 
the  relative  importance  of  applying  the  AFSCR  84-2  PRR 
questions  to  alleviate  some  of  the  confusion  in  the 
field  about  how  to  best  accomplish  a PRR  program  and 
where  to  expend  the  system  program  office  and  contract 
management  organization  resources  that  are  available  for 
conducting  a PRR  program. 

Corollary  Conclusions 

This  section  of  the  research  study  addresses 
the  corollary  conclusions  that  the  researchers  made 
from  the  six  corollary  findings  covered  in  the  previous 
chapter.  The  corollary  conclusions  are  the  interpre- 
tations of  the  researchers  from  their  studies  and  obser- 
vations while  performing  research  on  the  Air  Force  Sys- 
tems Command's  Production  Readiness  Review  (PRR)  tech- 
nique . 

As  indicated  in  the  first  corollary  finding, 
five  or  more  of  the  ten  AFSC  production  management  experts 
agreed  that  twenty  of  the  total  twenty-five  AFSCR  84-2 
PRR  questions  should  be  addressed  prior  to  DSARC  II. 
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The  initial  survey  indicated  that  a one-time  Production 
Readiness  Review  (PRR)  conducted  just  prior  to  DSARC  III 
will  not  suffice.  So,  the  conclusion  is  that  many  of  the 
AFSCR  84-2  PRR  questions  should  be  evaluated  long  before 
DSARC  III  to  insure  a successful  PRR  program.  Even 
though  the  twenty-five  AFSCR  84-2  PRR  questions  address 
the  readiness  of  a contractor  to  make  the  development- 
to-production  transition,  a system  program  office  should 
not  wait  until  the  end  of  the  full-scale  development 
phase  of  the  weapons  acquisition  process  to  start  the 
PRR  activity. 

The  second  and  third  corollary  findings  covered 
in  the  previous  chapter  address  the  same  major  problem 
area.  The  conclusion  from  these  two  corollary  findings 
is  that  the  present  guidance  available  for  the  system 
program  offices  that  are  tasked  to  conduct  PRR  programs 
is  not  sufficient.  The  present  AFSCR  84-2  is  incomplete 
and  should  be  modified  to  reflect  the  experience  or 
lessons  learned  from  the  PRR  programs  that  have  been  con- 
ducted since  the  regulation  was  published  in  November  1971. 
The  researchers  concluded  that  the  present  level  of 
production  management  staff  level  support  is,  in  the 
aggregate,  lacking  in  the  area  of  planning  for  and  imple- 
menting PRR  programs  for  major  Air  Force  weapon  systems. 

An  impact  is  that  some  critical  system  program  office 
and  other  governmental  office  resources  may  be  wasted  by 
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starting  from  the  very  beginning  for  each  new  PRR  program 
that  is  initiated. 

The  fourth  corollary  finding  addresses  the  costs 
versus  savings  relationship  that  the  Air  Force  should  be 
able  to  identify  to  determine  the  value  of  implementing 
PRR  programs.  The  interview  respondents  were  unable  to 
provide  cost  data  on  what  it  costs  the  Air  Force  to  con- 
duct PRR  programs,  and  the  interviewees  were  unable  to 
provide  any  documentation  on  potential  dollar  savings 
that  may  accrue  as  a result  of  conducting  a PRR  program 
on  a major  Air  Force  weapon  system.  It  is  concluded 
that  to  determine  the  value  and  benefits  derived  from 
PRR  programs,  the  Air  Force  must  be  able  to  obtain  some 
type  of  cost  data  and  corresponding  dollar  savings  data 
to  provide  a quantifiable  measure  of  the  worth  of  Air 
Force  PRR  programs. 

The  fifth  corollary  finding  pertains  to  the  fact 
that  some  interview  respondents  were  concerned  about  the 
lack  of  any  PRR  guidance  for  smaller  Air  Force  weapon 
system  programs  (i.e.,  those  weapon  system  programs  under 
$50  million  development  or  $200  million  production) . 

AFSCR  84-2  pertains  primarily  to  those  major  weapon  system 
programs  that  meet  the  criteria  established  in  DOD 
Directive  5000.1  (42:1).  The  researchers  concluded 
that  a portion  of  AFSCR  84-2  could  be  dedicated  to  the 
smaller  weapon  system  programs  in  which  an  abbreviated 
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PRR  approach  could  be  used  to  cover  the  key  production 
readiness  areas.  Preliminary  guidance  could  be  provided 
in  AFSCR  84-2  for  the  smaller  Air  Force  programs. 

The  final  corollary  finding  addressed  the  fact 
that  the  Air  Force  is  the  only  military  department  that 
has  a formalized  technique  to  assess  a major  weapon  system 
contractor's  readiness  to  transition  from  the  full- 
scale  development  phase  into  the  production  phase  of  the 
weapons  acquisition  process.  Since  the  Air  Force  PRR 
technique  evolved  from  DOD  level  concerns  about  the  lack 
of  emphasis  on  assuring  that  a contractor  is  ready  to 
make  the  development-to-production  transition,  the  con- 
clusion is  that  the  other  military  departments  should 
have  a similar  type  of  technique  to  assess  the  contractor's 
readiness  to  move  into  production.  To  date,  the  three 
major  Air  Force  weapon  system  programs  that  have  con- 
ducted PRR  programs  have  received  favorable  DSARC  III 
decisions.  Although  documented  savings  from  PRR  program 
efforts  are  not  available,  it  was  the  general  consensus 
of  the  AFSC  production  management  experts  that  the  benefits 
obtained  from  the  PRR  programs  far  exceeded  the  costs  of 
planning  and  implementing  the  PRR  programs. 


CHAPTER  VI 
RECOMMENDATIONS 
Overview 

As  noted  in  the  previous  chapters,  the  use  of 
the  PRR  technique  is  relatively  new,  and  application 
of  the  technique  is  primarily  concentrated  on  the  major 
Air  Force  weapon  system  programs . Although  only  three 
Air  Force  PRR  programs  have  been  completed  to  date, 
many  more  major  Air  Force  system  program  offices  will 
be  required  by  AFSCR  84-2  to  conduct  PRR  programs. 

For  example,  the  F-16  System  Program  Office  is  presently 
conducting  its  PRR  program.  Because  of  the  estimated 
contract  dollar  amounts,  the  Advanced  Medium  Short  Take- 
off and  Landing  Transports  (AMST)  buying  activity  should 
soon  begin  some  initial  PRR  program  planning  activity. 
During  this  study  effort  on  the  Air  Force  PRR  programs, 
two  specific  areas  of  recommendations  have  emerged. 

First,  some  specific  recommendations  for  implementation 
will  be  discussed;  and,  secondly,  some  recommendations 
for  future  research  will  be  provided  to  help  channel 
additional  research  efforts  in  this  area. 

Recommendations  for  Implementation 

As  a result  of  the  primary  findings  and  conclusions 

obtained  in  testing  Research  Hypotheses  No.  1 and  No.  2 
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and  the  corollary  findings  and  conclusions,  six  specific 
recommendations  are  presented  for  implementation  by  dif- 
ferent Air  Force  and  other  DOD  organizations. 

Recommendations  for 
Future  PRR  Programs 

The  findings  obtained  in  testing  Research  Hypoth- 
esis No.  1 were  not  generalized  outside  the  population 
of  the  three  major  Air  Force  PRR  programs  that  have  been 
completed.  But,  after  analyzing  the  data  collected  for 
testing  Research  Hypothesis  No.  2,  the  researchers  have 
attempted  to  provide  some  guidelines  for  planning  and 
implementing  PRR  programs  for  future  Air  Force  major  weapon 
systems.  The  action  offices  for  implementing  this 
particular  recommendation  will  be  those  Air  Force  system 
program  offices  and  Air  Force  Plant  Representative  Offices 
who  must  conduct  the  future  PRR  programs. 

The  researchers  recommend  that  future  PRR  programs 
be  developed  around  the  prioritized  list  of  twenty-five 
AFSCR  84-2  PRR  questions  furnished  in  Appendix  L.  The 
questions  at  the  top  of  the  prioritized  list  of  AFSCR 
84-2  PRR  questions  should  receive  the  most  emphasis  in 
future  PRR  programs;  the  lower-ranking  questions  on  the 
prioritized  list  should  receive  lesser  emphasis  in  future 
PRR  programs.  Future  PRR  program  organizers  should 
acknowledge  the  critical  importance  of  accomplishing  the 
top  six  of  the  twenty-five  AFSCR  84-2  PRR  questions.  The 
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future  PRR  program  planners  and  implementers  should  recog- 
nize that  a considerable  amount  of  PRR  program  effort 
should  be  spent  on  insuring  that  most  engineering  and 
testing  activities  have  been  accomplished  prior  to  move- 
ment into  the  production  phase  of  the  acquisition  process. 
This  recommendation  parallels  many  recent  suggestions 
of  different  panels  and  commissions  assigned  the  task 
of  evaluating  present  military  procurement  practices. 

To  implement  the  recommendation,  future  SPOs  and  AFPROs 
should  divide  the  prioritized  list  of  twenty-five  AFSCR 
84-2  PRR  questions  into  segments  that  can  be  covered 
thoroughly  in  incremental  visits  to  the  contractor.  The 
contractor  can  then  be  initially  approached  with  the  six 
high  priority  AFSCR  84-2  PRR  questions.  The  remaining 
nineteen  AFSCR  84-2  PRR  questions  can  be  broken  into 
sections  as  determined  by  the  individual  system  program 
office.  The  researchers  acknowledge  that  some  new  PRR 
evaluation  questions  may  be  generated  by  each  individual 
system  program  office.  These  new  questions  can  be  placed 
in  the  baseline  prioritized  list  as  determined  appro- 
priate for  the  individual  PRR  program. 

Another  specific  recommendation  for  future  PRR 
programs  is  that  initial  PRR  planning  and  evaluation 
activities  should  begin  as  early  as  DSARC  II.  Two  of 
the  three  major  weapon  system  program  offices  that  have 
completed  PRRs  to  date  spent  approximately  thirty  months 


on  the  PRR  program.  This  recommendation  is  based  on  the 
first  corollary  finding  in  which  the  AFSC  experts  in 
the  area  indicated  that  the  PRR  program  should  actually 
begin  early  in  the  weapons  acquisition  process  rather 
than  just  prior  to  DSARC  III.  An  incremental  approach 
is  recommended  for  the  PRR  program  in  which  the  entire 
PRR  program  effort  is  conducted  in  stages  that  will  coin- 
cide with  other  program  activities.  This  incremental 
approach  has  proven  successful  on  PRR  programs  that  have 
been  conducted  to  date . The  incremental  approach  can 
be  used  with  the  segmented  prioritized  list  of  AFSCR  84-2 
PRR  questions  mentioned  previously.  For  example,  the 
initial  set  of  AFSCR  84-2  PRR  questions  that  should  be 
evaluated  early  in  the  PRR  program  would  be  the  first 
six  of  the  prioritized  list  provided  in  Appendix  L. 

Based  on  the  primary  and  corollary  findings  and 
conclusions,  and  on  the  researchers'  interpretations  of 
the  study  results,  some  additional  factors  are  provided 
below  that  should  be  considered  in  future  PRR  programs : 

1.  The  system  program  office  (SPO)  should  insure 
that  adequate  contractual  coverage  is  secured  for  all 
contractors  (i.e.,  prime  contractor  and  subcontractors) 
who  will  be  involved  in  the  PRR  program. 

2 . A PRR  convention  or  preproduction  meeting 
should  be  held  early  enough  in  the  PRR  program  to  bring 
all  the  PRR  participants  together  to  discuss  the  PRR 


program  plan  and  the  specific  implementation  of  the  plan. 
The  convention  should  prevent  many  potential  problems 
downstream. 

3.  The  PRR  program  should  be  tailored  to  the 
individual  program  characteristics  using  the  prioritized 
list  of  AFSCR  84-2  PRR  questions  and  the  other  specific 
recommendations  above  as  a baseline  model. 

4.  If  subcontractor  PRRs  are  conducted,  a prime 
contractor  representative  should  be  in  attendance  at  each 
PRR  to  prevent  the  government  from  inadvertently  assuming 
a position  between  the  prime  contractor  and  his  subcon- 
tractors . 

5.  When  the  PRR  team  visits  a contractor,  the 

PRR  team  leader  should  make  explicit  remarks  in  the  initial 
and  exit  briefings  that  the  contractor  should  assume  no 
contractual  direction  from  the  comments  or  actions  of 
any  of  the  PRR  team  members.  All  direction  should  be 
given  by  the  Procuring  Contracting  Officer  to  the  prime 
contractor. 

6.  The  SPO  and  contract  management  organizations 
should  begin  PRR  planning  early  in  the  full-scale  develop- 
ment phase  of  the  weapons  acquisition  process  in  order 

to  complete  the  PRR  effort  prior  to  DSARC  III.  Sufficient 
time  should  be  allowed  for  schedule  slippages,  contractor 
revisits,  and  Department  of  Defense  Product  Engineering 
Services  Office  (PESO)  visits  (see  page  25) . 


95 

7.  The  Air  Force  Program  Manager  should  direct 
all  SPO  personnel  to  consider  their  support  to  the  PRR 
effort  as  being  a very  high  priority  task. 

8.  The  SPO  should  utilize  experienced  personnel 
(e.g.,  Air  Force  Materials  Laboratory  (AFML) , previous 
PRR  program  experts,  etc.)  to  conduct  a comprehensive  PRR 
of  the  weapon  system  program. 

9.  The  PRR  team  should  be  specifically  tailored 
to  evaluate  the  particular  contractors,  equipment,  manu- 
facturing processes,  etc.  under  review. 

10.  The  PRR  subject  should  be  addressed  early 
enough  in  the  weapons  acquisition  process  to  insure  that 
producibility  and  production  readiness  considerations 

are  included  in  engineering  preliminary  design  reviews  and 
critical  design  reviews. 

11.  A systematic  approach  should  be  developed 

to  handle  the  administrative  workload  caused  in  managing 
a PRR  program. 

12.  Government  communication  with  subcontractors 
concerning  PRR  program  activities  should  be  conducted 
through  the  prime  contractor  and  not  directly  to  the  sub- 
contractors . 

13.  The  SPO's  routine  communication  with  the 
secondary  delegated  contract  management  organizations 
should  be  conducted  through  the  prime  contract  management 
organization.  As  a minimum,  the  SPO  should  keep  the  prime 
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contract  management  office  informed  of  the  SPO  and  secon- 
dary contract  management  office's  activities. 


14.  The  SPO  should  document  PRR  program  activities. 
Background  documentation  that  is  used  as  a basis  for  the 
final  PRR  report  to  the  Air  Force  Program  Manager  should 
be  prepared  in  order  to  provide  backup  justification 

for  the  recommendations  included  in  the  final  PRR  report. 

15.  The  SPO  and  AFPRO  should  determine  the  avail- 
ability of  existing  data  sources  and/utilize  the  avail- 
able data  to  support  PRR  activities  (e.g.,  pre-award 
surveys,  AFCMD  Contractor  Management  System  Evaluation 
Program,  should  cost,  source^election  data,  etc.). 

AFSCR  84-2  Revision 

The  researcher^are  aware  of  the  present  Air  Force 
Systems  Command  activity  to  write  a pamphlet  that  will 
assist  Air  Force  system  program  offices  with  the  conduct 
of  the  future  MM/^CR  and  PRR  programs  (24).  But,  it  is 

presently  the  i/tent  of  Air  Force  Systems  Command  (AFSC) 

/ 

to  keep  the  current  AFSCR  84-2  intact.  The  researchers 
recommend  that,  on  the  basis  of  the  experience  of  three 
PRR  programs,  a rewrite  of  the  basic  regulation  is  in 
order.  For  example,  the  researchers  suggest  that  the 
twenty-five  AFSCR  84-2  PRR  questions  should  be  rearranged 
to  reflect  the  relative  importance  of  the  twenty-five 
AFSCR  84-2  PRR  questions  to  help  insure  a successful 
PRR  program.  The  present  unorganized  list  of  the 
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AFSCR  84-2  questions  could  be  eliminated,  thereby  poten- 
tially reducing  many  of  the  initial  hurdles  in  developing 
a PRR  program.  Another  major  suggestion  that  should  be 
incorporated  into  a revised  AFSCR  84-2  is  the  fact  that 
a PRR  should  not  be  a one-time  activity  held  just  prior 
to  DSARC  III,  but,  instead,  a PRR  should  be  a program  of 
activity  that  should  begin  in  the  full-scale  development 
phase  and  should  lead  up  to  the  DSARC  III  decision- 
making process.  So,  the  researchers  recommend  for  the 
Director  of  Manufacturing,  DCS/Procurement  and  Manu- 
facturing, Air  Force  Systems  Command  to  rewrite  the  basic 
regulation  on  the  Air  Force  Production  Readiness  Review 
(AFSCR  84-2) . 

More  Production  Staff 
Involvement 

At  present  there  is  no  central  repository  of 
information  or  expertise  on  how  various  PRR  programs  have 
been  planned  and  conducted  to  date.  The  information  can 
be  obtained  only  by  contacting  each  major  system  program 
office  separately  that  has  conducted  PRR  programs  to  date. 
With  the  turnover  of  personnel  in  the  SPO  organizations, 
some  valuable  information  and  experience  are  probably 
being  lost.  The  researchers  recommend  that  increased 
emphasis  should  be  placed  by  staff  level  organizations 
(i.e.,  ASD  staff,  ESD  staff,  and  AFSC  staff)  on  the  les- 
sons learned  from  previous  PRR  programs.  At  least  one 
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individual  in  each  staff  level  organization  should  be 
tasked  with  the  responsibility  for  collecting  information 
on  the  lessons  learned  from  previous  PRR  programs  for 
dissemination  to  the  field  organizations  upon  request. 

By  centralizing  this  responsibility  in  production  staff 
offices,  any  system  program  office  required  to  conduct 
a PRR  program  in  the  future  can  utilize  the  expertise  and 
request  the  assistance  that  will  be  available  in  the 
staff  level  organizations.  It  is  recommended  that  overall 
responsibility  for  developing  the  central  repository 
of  PRR  information  and  expertise  be  placed  upon  the 
Director  of  Manufacturing,  DCS/Procurement  and  Manufac- 
turing, Air  Force  Systems  Command  (AFSC) . 

Emphasize  Costs  Versus  Benefits 

A major  problem  that  confronted  the  researchers 
was  the  lack  of  documented  costs  and  savings  data  on 
the  three  major  Air  Force  PRR  programs  conducted  to  date. 

A common  trend  in  instituting  new  programs  or  techniques 
in  the  past  has  been  blind  adherence  without  trying  to 
determine  the  cost  versus  benefit  relationship  between 
conducting  new  programs  and  the  potential  savings  that 
may  be  realized  from  the  new  programs.  So,  the  researchers 
recommend  that  system  program  offices  and  other  govern- 
ment organizations  required  to  conduct  PRR  programs 
document  costs  and  potential  savings  in  an  effort  to 
determine  the  value  of  the  Air  Force  PRR  technique. 


Based  on  interviews  with  system  program  office  personnel, 
it  is  concluded  that  cost  and  savings  documentation  is 
feasible  and  should  produce  some  results  indicating  the 
usefulness  of  the  Air  Force  PRR  technique. 

Mini-PRR  Program  Plan 

The  present  AFSCR  84-2  applies  to  those  major 
programs  that  meet  the  criteria  of  DOD  Directive  5000.1 
(42:1).  For  those  Air  Force  weapon  system  programs 
which  do  not  meet  the  criteria  for  classification  as  a 
major  program,  the  guidance  for  determining  if  a contrac- 
tor is  ready  to  produce  on  a contract  is  somewhat  limited 
The  researchers  recommend  a study  to  investigate  the 
feasibility  of  developing  a mini-PRR  program  plan  for  use 
by  the  smaller  programs  that  are  required  to  determine  a 
contractor's  readiness  to  produce.  This  recommended 
study  should  be  performed  by  any  one  of  the  ASD,  BSD, 
or  AFSC  production  management  staff  level  organizations. 
It  may  be  feasible  to  incorporate  a mini-PRR  plan  into 
the  recommended  revision  of  AFSCR  84-2.  At  a minimum, 
the  first  six  of  the  prioritized  list  of  AFSCR  84-2  PRR 
questions  (Appendix  L)  should  be  reviewed  foi  the  smaller 
programs.  By  implementing  this  recommendation,  the 
smaller  system  program  offices  will  obtain  some  needed 
guidance  on  what  should  be  done  for  the  smaller  Air  Force 
programs  to  protect  the  interests  of  the  government  and 
the  taxpayers. 
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Expanded  POD  Use  of 
PRR  Technique 

As  of  the  writing  of  this  study  report,  the  Air 
Force  is  the  only  service  to  have  developed  a formalized 
PRR  procedure  to  assess  a contractor's  readiness  to 
make  the  transition  from  the  full-scale  development  phase 
to  the  production  phase  of  the  weapons  acquisition  process 
for  major  programs.  The  experience  on  the  three  Air 
Force  programs — F-15,  A-10,  and  AWACS — that  have  completed 
PRR  programs  has  been  encouraging.  All  three  major  weapon 
system  programs  received  favorable  DSARC  III  decisions 
and  were  authorized  to  proceed  into  the  production  phase 
of  the  weapons  acquisition  process.  The  interview  respon- 
dents were  unanimous  in  agreeing  that  the  PRR  technique 
is  beneficial  and  should  help  prevent  many  potential  prob- 
lems in  the  production  phase  of  the  weapons  acquisition 
process.  Due  to  the  critical  nature  of  the  development- 
to-production  transition  process  and  the  large  amount  of 
taxpayers' dollars  that  will  be  obligated  when  production 
go-ahead  is  authorized,  it  is  recommended  for  the  other 
military  services  to  develop  similar  Production  Readiness 
Review  (PRR)  procedures  to  insure  that  a contractor  is 
ready  to  go  into  production.  Some  general  DOD  guidance 
in  this  area  will  facilitate  implementation  of  this 


recommendation . 
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Recommendations  for  Future  Research 

As  discussed  previously,  the  use  of  the  PRR 
technique  is  relatively  new,  and  application  of  the  tech- 
nique has  been  relatively  limited.  But,  as  new  major 
weapon  systems  are  developed,  the  requirement  to  conduct 
a PRR  program  will  prevail.  The  B-l  System  Program  Office 
has  just  completed  much  of  its  PRR  program  activities  (6) . 
The  F-16  System  Program  Office  is  presently  in  the  midst 
of  conducting  a comprehensive  PRR  program  (10)  , and  the 
AMST  program  will  soon  begin  the  initial  PRR  program  plan- 
ning activities.  Much  is  yet  to  be  learned  about  the 
usefulness  and  total  impact  of  the  PRR  technique.  With 
the  increase  in  the  number  of  PRR  programs  available  for 
study,  the  possibilities  for  additional  meaningful 
research  on  the  Air  Force  PRR  technique  will  be  enhanced. 
This  section  briefly  discusses  five  specific  potential 
areas  for  future  research. 

Replication  of  This  Study 

Research  on  the  three  major  PRR  programs  conducted 
to  date  and  interviews  with  the  ten  Air  Force  Systems 
Command  (AFSC)  production  management  experts  on  the  rela- 
tive importance  of  evaluating  certain  features  of  a 
contractor's  production  readiness  do  not  present  conclu- 
sive evidence  that  the  trends  of  the  findings  can  be 
inferred  to  apply  to  all  AFSC  major  weapon  system  programs. 
But,  the  methodology  used  in  the  instant  study  was 
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validated  and  the  research  procedures  have  been  well- 
documented  to  allow  for  replication  of  this  study.  As 
additional  experience  is  gained  in  conducting  effective 
and  efficient  PRR  programs,  the  potential  for  future 
meaningful  research  in  this  area  is  great.  Therefore, 
in  an  effort  to  establish  greater  confidence  in  the 
research  results,  replication  is  recommended  with 
emphasis  on  other  ASO,  ESD,  and  SAMSO  programs. 

Contractor  Viewpoint  Analysis 

Since  this  study  has  addressed  the  Air  Force  PRR 
technique  and  its  effectiveness  from  the  government  side 
only,  the  conclusions  reached  may  be  one-sided.  One  way 
to  determine  the  usefulness  and  effectiveness  of  the  PRR 
technique  on  evaluating  a contractor's  readiness  to  make 
the  full-scale  development-to-production  transition  would 
be  to  determine  contractors'  perceptions  of  the  present 
PRR  technique's  usefulness  and  effectiveness.  The  con- 
tractors for  the  A-10,  F-15,  and  AWACS  programs  could  be 
surveyed  along  with  any  additional  contractors  that  have 
been  involved  in  Air  Force  PRR  activities. 

Cost  Versus  Benefit  Study 

As  disclosed  in  previous  chapters,  the  background 
literature  indicated  an  absence  of  information  on  compari- 
sons of  total  PRR  program  costs  versus  the  benefits  gained 
from  the  PRR  program  activity.  A study  of  the  cost  versus 
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benefit  relationship  could  concentrate  on  those  programs 
which  have  documented  some  cost  figures  and  have  attempted 
to  identify  the  monetary  benefits  accrued  as  a direct 
result  of  PRR  program  activities. 

Pricing  for  PRR  Requirement 

Very  little  information  has  been  printed  on  the 
contractor's  pricing  of  PRR  program  requirements.  Some 
contractors  contend  that  no  contract  price  increase 
resulted  from  the  PRR  requirement  (49),  while  other  con- 
tractors are  specifically  identifying  contract  line  items 
with  respective  cost  information  for  meeting  Air  Force 
PRR  program  requirements  (6) . The  contractor  methodology 
used  in  identifying  the  factors  considered  in  arriving 
at  the  pricing  figures  has  not  been  explicitly  stated. 

A study  should  be  conducted  to  identify  potential  factors 
to  be  considered  in  arriving  at  a fair  and  reasonable  price 
for  satisfying  Air  Force  PRR  program  requirements. 

PRR  Responsibilities 
and  Functions 

The  two  primary  organizations  that  are  involved 
in  the  conduct  of  a PRR  program  are  the  system  program 
office  and  the  respective  prime  contract  administration 
organization.  It  may  be  that  some  PRR  program  activities 
and  AFSCR  84-2  PRR  questions  can  be  more  efficiently  and 
effectively  handled  by  one  of  these  organizations  (20) . 
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The  researchers  recommend  a study  to  investigate  and  clas- 
sify the  major  responsibilities,  functions,  and  tasks 
which  properly  should  be  assigned  to  the  respective  SPO 
or  contract  administration  organizations  in  the  conduct 
of  PRR  programs. 

Concluding  Observations 

After  research  in  analyzing  the  Air  Force's  PRR 
technique,  the  authors  conclude  that  the  PRR  technique 
has  merit  for  several  reasons.  First,  the  Air  Force  can 
be  more  assured  that  planned  major  weapon  systems  will 
receive  a positive  DSARC  III  and  Secretary  of  Defense 
approval  on  full  production  go-ahead.  Three  major  Air 
Force  weapon  system  programs — A-10,  F-15,  and  AWACS — 
that  have  completed  a PRR  program  have  received  production 
authorizations.  Second,  after  completing  a PRR  program 
effort,  the  Air  Force  buying  activity  responsible  for  a 
major  weapon  system  can  feel  more  confident  that  a con- 
tractor is  ready  to  make  the  development-to-production 
transition.  The  impact  of  this  is  that  many  potential 
problem  areas  will  be  corrected  early  in  the  weapons 
acquisition  process,  and  the  possibility  of  a major 
weapon  system  procurement  fiasco  will  be  reduced.  And 
finally,  there  will  be  more  Air  Force  and  contractor 
interaction  and  involvement  early  in  the  weapons  acquisi- 
tion process  to  guard  against  potential  problems  that 
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could  be  encountered  in  the  production  phase  of  the  weapons 
acquisition  process.  The  United  States  taxpayers  will 
be  the  primary  beneficiaries  of  the  PRR  program  activi- 
ties. 

While  much  has  been  written  on  the  need  for  insur- 
ing that  contractors  are  ready  to  transition  from  the  full- 
scale  development  phase  into  the  production  phase  of  the 
weapons  acquisition  process,  very  little  research  is  found 
in  this  area  of  procurement.  Since  there  are  areas  needing 
additional  investigation,  it  is  hoped  that  the  study  will 
serve  as  a catalyst  for  further  examinations  of  the  PRR 
technique's  application  and  usefulness  in  the  weapons 
acquisition  process. 
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APPENDIX  A 


INTERVIEW  GUIDE  NO.  1 


1+  How  many  PRR  visits  were  made  to  the  prime  contractor's 
facility? 

PRR  visits  to  prime  contractor  are  the  total  number 
of  vi8it8  that  a system  program  office  PRR  team  of  any 
size  made  to  review  the  production  readiness  of  the 
prime  contractor  using  the  AFSCR  84-2  questions  as 
guidance . 

Response 


2 How  many  PRR  visits  were  made  to  subcontractors? 

PRR  visits  to  subcontractors  are  the  total  number 
of  visits  that  an  entire  PRR  team  of  any  size  made  to 
review  the  production  readiness  of  any  subcontractor 
using  AFSCR  84-2  as  guidance . 

Response 


3t  How  many  different  skills  (SPO)  were  used  in  the  PRR 
program? 

Different  skills  (SPO)  are  the  total  number  of  func- 
tional skills  represented  in  the  system  program 
office's  PRR  visits  to  any  contractor.  Functional 
skill  is  each  different  job  classification  as  deter- 
mined by  an  individual's  position  description. 

Response 


4t  How  many  PRR  team  members  (SPO)  were  involved  in  the 
PRR  program? 

PRR  team  members  (SPO)  are  the  total  number  of  visits 
to  any  contractor  made  by  all  personnel  from  the  system 
program  office  as  a participant  on  a PRR  team  using 
AFSCR  84-2  as  a guide.  Each  visit  made  by  each  SPO  PRR 
team  member  to  any  contractor  will  be  counted  as  one 
PRR  team  member  (SPO). 

Response 
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5*  How  many  different  skills  (AFPRO)  were  used  in  the  PRR 
program? 

Different  skills  (AFPRO)  are  the  total  number  of  func- 
tional skills  represented  from  the  local  Air  Force 
Plant  Representative  Office  (AFPRO)  at  all  PRR 
meetings  held  with  the  prime  contractor  or  any  sub- 
contractors using  AFSCR  84-2  as  guidance.  Functional 
skill  is  each  different  job  classification  as  deter- 
mined by  an  individual  ' s position  description. 

Response 


6*  How  many  PRR  team  members  (AFPRO)  were  involved  in 
the  PRR  program? 

PRR  team  members  (AFPRO)  are  the  total  number  of 
times  all  Air  Force  Plant  Representative  Office 
personnel  participated  as  a PRR  team  member  on  any 
review  held  at  the  prime  contractor's  facility  or  on 
any  visit  to  a subcontractor  using  AFSCR  84-2  as 
guidance . Each  different  visit  or  participation  for 
each  PRR  team  member  will  be  counted  as  one  PRR  team 
member  (AFPRO). 

Response 


7 How  many  supplemental  PRR  team  members  were  involved 
in  the  PRR  program? 

Supplemental  PRR  team  members  are  the  total  number 
of  visits  to  any  contractor  made  by  all  personnel 
not  in  the  system  program  office  or  AFPRO  to  partici- 
pate on  a PRR  team  with  AFSCR  84-2  as  guidance . Each 
visit  made  by  each  supplemental  PRR  team  member  to 
any  contractor  will  be  counted  as  one  supplemental 
PRR  team  member. 

Response 


8t  What  was  the  duration  of  the  PRR  program? 

Duration  of  PRR  program  is  the  total  amount  of  time 
(in  months)  from  the  start  of  the  planning  phase  of 
the  PRR  program  up  to  the  time  of  the  DSARC  III 
review.  The  start  of  the  PRR  planning  phase  is  when 
the  system  program  office  actually  began  working  on 
the  PRR  program. 


Response 
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9 How  many  verbatim  AFSCR  84-2  PRR  questions  were  used 
in  the  PRR  program? 

Verbatim  AFSCR  84-2  PRR  questions  are  the  total  number 
of  the  twenty- five  standard  AFSCR  84-2  PRR  questions 
that  were  used  on  any  PRR  visit  for  the  entire  PRR 
program.  Each  of  the  twenty- five  questions  will  be 
counted  only  once. 


Response 


10  How  many  additional  PRR  questions  were  used  in  the  PRR 
program? 

Additional  PRR  questions  are  the  total  number  of 
questions  used  in  the  PRR  program  that  were  either 
modified  AFSCR  84-2  questions  or  new  questions  addres- 
sing other  areas  not  covered  by  the  AFSCR  84-2  ques- 
tions. Each  additional  PRR  question  will  be  counted 
only  once. 

Response 


| lit  How  much  did  the  PRR  program  cost  the  SPO? 

PRR  program  cost  (SPO)  is  the  total  estimated  cost 
incurred  by  the  system  program  office  in  conducting 
the  entire  PRR  program.  The  estimate  will  include 
PRR  contractual  coverage  expenses , temporary  duty 
travel  expenses,  typing  expenses , report  publication 
expenses , manpower  costs,  etc.  that  can  be  directly 
attributed  to  the  conduct  of  the  entire  PRR  program. 

Response 


• 12*  How  much  did  the  PRR  program  cost  the  AFPRO? 

PRR  program  cost  (AFPRO)  is  the  total  estimated  cost 
incurred  by  the  AFPRO  in  conducting  the  entire  PRR  pro- 
gram. The  estimate  will  include  temporary  duty  travel 
expenses , typing  expenses , report  publication  expenses , 
manpower  costs,  etc.  that  can  be  directly  attributed  to 
the  conduct  of  the  entire  PRR  program. 

Response 


* 


*For  AFPRO  interviews  only.  +For  SPO  interviews  only. 
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APPENDIX  B 

STANDARD  AFSCR  84-2  PRR  QUESTIONS  (42:5-6) 

1.  Milestones  which  demonstrate  the  achievement 
of  a practical  and  producible  engineering  design  have 
been  met. 

2.  Engineering  problems  encountered  during  develop- 
ment have  been  resolved  with  appropriate  trade-offs 
against  stated  operating  requirements  so  that  production 
costs/schedules  are  optimized. 

3.  Critical  production  engineering  and  production 
tooling  have  been  demonstrated  to  prove  that  engineering 
has  been  satisfactorily  accomplished. 

4.  Acquisition  will  smoothly  transition  from  full- 
scale  development  to  production. 

5.  System  configuration  has  been  reviewed  to 
determine  if  any  significant  design  changes  will  be 
required  for  manufacturing. 

6.  Adequate  advanced  production  planning  has  been 
accomplished  and  required  production  controls  established 
to  ensure  timely  production. 

7.  A systematic  approach  to  standardization  has 
been  accomplished  in  the  design  process  and  parts  selection 
to  maximize  the  use  of  military  standard  components,  parts, 
and  processes  consistent  with  the  system  requirements. 
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8.  Product  assurance  controls  and  tests  to  prevent 
manufacturing  degradation  of  performance  parameters  have 
been  established. 

9.  Assurance  of  readiness  of  the  manufacturing 
and  production  equipment,  methods,  facilities,  test  and 
training  equipment,  and  status  of  accessory  and  ancillary 
items . 

10.  Planned  production  schedules  reflect  economy 
of  operations  and  minimize  financial  commitments  until  all 
major  development  problems  have  been  resolved. 

11.  A thorough  assessment  of  the  make-or-buy 
structure  has  been  accomplished  and  procedures  exist  so 
control  and  visibility  of  the  vendors  and  subcontractors 
can  be  effectively  maintained. 

12.  Change  activity  during  development  has  been 
evaluated  and  the  impact  of  outstanding  changes  on  pro- 
duction has  been  assessed. 

13.  Results  of  technical  reviews  and  the  production 
impact  of  unresolved  problems  and  risk  have  been  assessed. 

14.  Test  program  results  and  the  status  of  quali- 
fication testing  to  determine  production  impact  and  risk 
have  been  evaluated. 

15.  Specifications  and  drawings  have  been  reviewed 
to  assure  their  adequacy  for  the  planned  production  phase. 


16.  Application  of  production  tooling  and  test 
equipment  to  manufacturing  during  development  has  been 


1X4 

assessed  and  the  application  of  same  to  the  production 
phase  has  been  defined. 

17.  Material  management  system  for  determination 
of  requirements,  procurement,  receiving,  inspection, 
materials  handling  and  storage,  inventory  control,  control 
of  finished  goods,  and  shipment  is  adequate. 

18.  Production  management  systems  used  for  pro- 
viding management  with  timely  production  status  informa- 
tion are  effective. 

19.  Production  or  manufacturing  capabilities  of 
major  subcontractors  and  vendors  have  been  technically 
evaluated  and  found  adequate. 

20.  Constraints  of  laboratory  or  model  shop  capa- 
bilities versus  quantity  production  requirements  have  been 
fully  considered. 

21.  Quality  controls  and  inspection  procedures 
have  been  established  for  materials  treatment  or  processes 
to  be  used  in  production. 

22.  Assessment  of  the  GFP  or  services  requirements, 
controls,  management,  and  availability  of  suppliers  has 
been  accomplished. 

23.  Availability  of  production  labor  skill  require- 
ments has  been  assessed  and  their  acquisition  adequately 
planned. 


24.  The  contractor  is  adequately  organized  to 
accomplish  the  production  requirements. 

25.  Planning  has  been  made  to  assure  timely 
release  of  manufacturing  instructions. 
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METHODOLOGY  FOR  CODING  DATA  FOR 
RESEARCH  HYPOTHESIS  NO.  1 

The  data  collected  to  test  Research  Hypothesis 
No.  1 were  coded  to  provide  a more  meaningful  compari- 
son of  the  three  major  weapon  system  PRR  programs  that 
have  been  completed  to  date.  By  coding  the  data  for 
Research  Hypothesis  No.  1,  it  was  possible  to  objectively 
compare  the  variability  between  the  high  data  values  and 
the  low  data  values  for  each  variable.  Variability, 
which  is  defined  as  a percentage  above  some  base  figure 
(lowest  variable  value),  should  not  be  confused  with 
variance  which  is  a measure  of  dispersion  about  the  mean. 

The  formula  used  to  code  the  twelve  variable 
values  was  as  follows: 


CV 


AV-LV 

LV 


where : 

CV  ■ The  coded  variability  value  (variability 
expressed  as  a decimal  instead  of  as  a 
percentage)  for  each  variable  for  each  PRR 
program. 
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AV  * The  actual  value  obtained  for  each  variable 
for  each  PRR  program. 

LV  * The  lowest  value  obtained  for  each  variable . 

To  illustrate  the  coding  procedure,  an  example 
is  provided.  The  number  of  PRR  visits  made  to  the  prime 
contractor  for  the  F-15,  A-10,  and  AWACS  programs  were 
27,  25,  and  3 respectively.  So,  the  actual  value  (AV) 
for  the  F-15  PRR  program  for  the  first  variable  was  27, 
and  the  lowest  value  (LV)  for  the  variable  was  3.  The 
coded  variability  value  calculation  for  the  F-15  PRR 
program  is  shown  below: 

cv  = mz 


cv 


27-3  24 

= T 


8.000 


The  coded  variability  value  (CV)  for  the  A-10  program 
for  the  first  variable  was  7.333: 


CV  « * 7.333 


The  CV  for  the  AWACS  program  for  the  first  variable 
was  0.000: 


CV  » 


3-3 


0.000 
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After  computing  CV  for  the  three  programs  (F-15, 
A-10,  and  AWACS)  for  each  of  the  twelve  variables  used 
for  testing  Research  Hypothesis  No.  1,  an  average  coded 
variability  value  was  calculated  for  each  of  the  twelve 
variables.  To  compute  the  average  coded  variability 
(CV)  for  each  variable,  the  following  formula  was  used: 

n 

cv  = £cv.. 

i=l 

n 

where : 

CV^  ■ The  coded  variability  value  for  each  one 
of  the  twelve  variables  for  the  F-15, 

A-10,  and  AWACS  programs, 
n = The  number  of  PRR  programs  completed  to 
date  (n=3) . 

CV  * The  average  coded  variability  value  for 
each  variable. 

Continuing  with  the  example  above,  the  CV  for 
the  "PRR  visits  to  prime  contractor"  variable  was  5.111. 
The  computations  are  shown  below: 

CV1  - 8.000 

CV2  - 7.333 

cv3  ■ 0.000 
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n = 3 
n 

l CV. 

cv  = i-1  1 = (8.000+7.333+0.000)  = s ni 

n 3 


An  average  coded  variability  value  was  calculated  for 
each  one  of  the  twelve  variables  used  to  test  Research 
Hypothesis  No.  1.  It  should  be  noted  that  for  each  vari- 
able having  an  actual  value  of  zero  a "one"  was  substi- 
tuted for  zero  so  that  the  CV  procedure  could  be  used 
for  all  twelve  variables.  Because  of  the  magnitude  of 
the  other  variable  values  for  the  two  instances  where 
this  occurred,  the  substitution  did  not  significantly 
impact  the  results  of  the  CV  analysis  technique. 
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APPENDIX  D 


INTERVIEW  GUIDE  NO.  2 
Ranking  Interview  Procedure 


A.  The  interviewee  will  be  given  a copy  of  the 
AFSCR  84-2  PRR  questions  (Appendix  B)  and  will  be  given 
sufficient  time  to  rank-order  the  twenty-five  AFSCR  84-2 
questions. 

B.  The  following  specific  instructions  will  be 
given  to  the  AFSC  production  management  expert: 

To  determine  if  a standard  PRR  approach  can 
be  developed  to  be  universally  and  practically 
applied  to  different  AFSC  weapon  system  programs , 
rank  the  twenty- five  standard  PRR  questions  in 
AFSCR  84-2  in  order  of  importance . A "one" 
should  be  assigned  to  the  PRR  question  that  should 
have  the  greatest  amount  of  time  and  resources 
expended  for  analysis  and  reporting , and  a 
"twenty- five"  should  be  assigned  to  the  least 
important  question. 

C.  The  interviewer  will  record  the  rankings  in 
the  attached  Table  3. 
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SOURCES  OF  EXPERT  OPINION  FOR 
RESEARCH  HYPOTHESIS  NO.  2 


APPENDIX  E 


SOURCES  OF  EXPERT  OPINION  FOR 
RESEARCH  HYPOTHESIS  NO.  2 


Ten  Air  Force  production  management  experts  were 
surveyed  to  provide  their  expert  opinion  on  the  relative 
importance  of  the  twenty-five  AFSCR  84-2  PRR  questions 
for  testing  Research  Hypothesis  No.  2.  Each  one  of  the 
production  management  experts  possessed  the  following 
minimum  qualifications:  (1)  active  in  the  production 

management  area  for  at  least  three  years;  (2)  involved 
in  the  management  of  major  Air  Force  weapon  systems  for 
at  least  five  years,  and  (3)  involved  in  the  planning 
or  implementation  of  at  least  one  AFSC  PRR  program.  The 
key  determinant  for  selecting  the  production  management 
experts  was  their  previous  PRR  experience. 

The  ten  production  management  experts  are  listed 

below: 


Lieutenant  Colonel  Jack  B.  Bryan,  USAF 
Chief  of  Production  Management  Branch 
Deputy  for  F-15 

Aeronautical  Systems  Division  (AFSC) 
Wright-Patterson  AFB,  Ohio 

Mr.  Thomas  L.  Campbell 
Supervisory  Industrial  Specialist 
Directorate  of  Production/Manufacturing 
Aeronautical  Systems  Division  (AFSC) 
Wright-Patterson  AFB,  Ohio 
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Major  Malcolm  C.  Edelblute,  USAF 
Chief  of  Production  Support  Division 
Deputy  for  B-l 

Aeronautical  Systems  Division  (AFSC) 

Wright-Patterson  AFB , Ohio 

Major  Ronald  S.  Joyner,  USAF,  Chief  of  Manufacturing 
Management  Division,  Deputy  for  A-10 
Aeronautical  Systems  Division  (AFSC) 

Wright-Patterson  AFB,  Ohio 

Major  Lyle  W.  Lockwood,  USAF 
Research  Associate 

Air  Force  Business  Research  Management  Center 
(HQ  USAF) 

Wright-Patterson  AFB,  Ohio 

Major  Ronald  D.  Morris,  USAF 
Manufacturing  Operations  Division  Chief 
Detachment  9 
AFCMD  (AFSC) 

The  Boeing  Company 
Seattle,  Washington 

Lieutenant  Colonel  David  E.  Otteson,  USAF 
Chief  of  Production  Division 
Deputy  for  F-15 

Aeronautical  Systems  Division  (AFSC) 

Wright-Patterson  AFB,  Ohio 

Captain  Leo  Pavlow,  USAF 
Manufacturing  Operations  Division  Chief 
Detachment  44 
AFCMD  (AFSC) 

Fairchild  Corporation 
Farmingdale,  L.I.,  New  York 

Captain  Stanley  Vlasak,  USAF 
Production  Engineer,  Production  Division 
Deputy  for  Airborne  Warning  and  Control  System 
(AWACS) 

Electronic  Systems  Division  (AFSC) 

Hanscom  AFB , Massachusetts 

Captain  David  S.  Wayman,  USAF 
Contracting  Officer 

Surveillance,  Navigation  and  Command  and  Management 
Systems 

Deputy  for  Procurement  and  Manufacturing 
Electronic  Systems  Division  (AFSC) 

Hanscom  AFB,  Massachusetts 
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APPENDIX  F 


Variables  for  Research  Hypothesis  No.  1 — Continued 
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APPENDIX  G 

CODED  VARIABILITY  OF  TWELVE  VARIABLES  FOR 
TESTING  RESEARCH  HYPOTHESIS  NO.  1 
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APPENDIX  G 


CODED  VARIABILITY  OF  TWELVE  VARIABLES  FOR 
TESTING  RESEARCH  HYPOTHESIS  NO.  1 


Variable 


Variability 
Above  Lowest 
Value 


Average 

Variability 


PRR  Visits  to  Prime 
Contractor 


8.000 

7.333 

0.000 


5.111 


PRR  Visits  to  Subcontractors 


4.000 

0.000 

0.800 

4.800 


1.600 


Different  Skills  (SPO) 


0.000 

1.000 

1.600 


0.867 


PRR  Team  Members  (SPO) 


2.500 

2.788 

0.000 

5:T88 


1.763 


Different  Skills  (AFPRO) 


0.250 

0.000 

1.750 

2.000 


0.667 


PRR  Team  Members  (AFPRO) 


0.000 

2.056 

0.417 

27573 


0.824 


Supplemental  PRR  Team  Members 


0.000 

54.000 

65.000 

IT575W 


39.667 
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Variable 

Variability 
Above  Lowest 
Value 

Average 

Variability 

Duration  of  PRR  Program 

0.000 

5.000 

4.800 

$.866 

3.267 

Verbatim  AFSCR  84-2  PRR 

Questions 

19.000 
0.000 

20.000 

39.000 

13.000 

Additional  PRR  Questions 

0.000 

49.000 

1.000 

sy.ooo 

16.667 

PRR  Program  Cost  (SPO) 

2.269 

8.615 

0.000 

16'.  3*4 

3.628 

PRR  Program  Cost  (AFPRO) 

0.000 

2.333 

9.000 

11.333 

3.778 

APPENDIX  H 

AFSC  PRODUCTION  MANAGEMENT  EXPERT  RANKINGS  FOR  THE 
TWENTY-FIVE  STANDARD  AFSCR  84-2 
PRR  QUESTIONS 
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APPENDIX  I 


THE  KENDALL  COEFFICIENT  OF  CONCORDANCE  W 
SUMMARY  PROCEDURE 

1.  Let  N equal  the  number  of  entities  to  be  ranked 
(25) , and  let  k equal  the  number  of  experts  assigning 
ranks  (10) . Arrange  the  observations  in  a k by  N 
matrix. 

2.  For  each  entity  (AFSCR  84-2  PRR  question), 
determine  i? . which  is  the  sum  of  the  ranks  assigned  to 
each  question  from  the  ten  experts. 

3.  Determine  the  mean  of  the  Ry  Express  each 

R.  as  a deviation  from  that  mean.  Square  these  devia- 

«7 

tions,  and  sum  the  squares  to  obtain  s. 

4.  Compute  W from  the  following  formula: 

w - I -;-  3 

YjJc  (N  -N) 


5.  The  method  of  determining  whether  the  calcu- 
lated value  of  W is  significantly  different  from  zero 
depends  on  the  size  of  N.  If  N is  larger  than  seven,  use 
the  formula : 
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OMNITAB  II  OUTPUT  FOR  ONE-WAY 
ANALYSIS  OF  VARIANCE 


Source 

df 

Sum  of 
Squares 

Mean 

Squares 

F 

Ratio 

F 

Prob 

Between 

Groups 

24 

3 . 261C00E 

03 

1 . 358750E 

02 

3.139 

0.000 

Slope 

1 

1. 331840E 

03 

1 . 331840E 

03 

28.307 

0.000 

Deviations 

About 

Line 

23 

1 . 929160E 

03 

8 . 387651E 

01 

1.938 

0.008 

Within 

- 

Groups 

225 

9 . 739000E 

03 

4 . 328444E 

01 

Total 

249 

1 . 300000E 

04 

140 
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COMPARATIVE  RANKINGS  OF  AFSCR  84-2 
NUMBERS  USING  MEDIANS  AND 


PRR  QUESTION 
MEANS 
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APPENDIX  K 

COMPARATIVE  RANKINGS  OF  AFSCR  84-2  PRR  QUESTION 
NUMBERS  USING  MEDIANS  AND  MEANS 


Ranking 

AFSCR  84-2 

PRR  Question  Numbers 

Using  Medians 

Using  Means 

1 

1 

6 

2 

6,13 

2 

3 

• • • • • 

1 

4 

2 

13 

5 

5 

5 

6 

14 

14 

7 

19 

19 

8 

4 

9 

9 

9 

3 

10 

3 

4,10 

11 

10 

12 

11,12,15 

12 

13 

15 

14 

• • • • • 

11 

15 

25 

21 

16 

21 

25 

17 

16,24* 

16 

18 

• • • • • 

18* 

19 

7,8* 

17* 

20 

• • • • • 

7 

21 

18* 

20,22 

22 

CM 

CM 

O 

CM 

23 

• • • • • 

24* 

24 

17* 

23 

25 

23 

8* 

•Ranking  differences  are  greater  than  two  steps 
depending  on  whether  means  or  medians  are  used  for  the 
rankings  (Questions  8,  17,  18,  24). 
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AGGREGATE  PRIORITY  LIST  OF  TWENTY-FIVE 
AFSCR  84-2  PRR  QUESTIONS  FROM  TEN  AFSC 
PRODUCTION  MANAGEMENT  EXPERTS 


Ranking 

Current 

Question 

Number 

AFSCR  84-2  PRR  Question 

1 

6 

Adequate  advanced  production  planning 
has  been  accomplished  and  required 
production  controls  established  to 
ensure  timely  production. 

2 

2 

Engineering  problems  encountered 
during  development  have  been  resolved 
with  appropriate  trade-offs  against 
stated  operating  requirements  so  that 
production  costs/schedules  are 
optimized. 

3 

1 

Milestones  which  demonstrate  the 
achievement  of  a practical  and  pro- 
ducible engineering  design  have  been 
met. 

4 

13 

Results  of  technical  reviews  and  the 
production  impact  of  unresolved  prob- 
lems and  risk  have  been  assessed. 

5 

5 

System  configuration  has  been  reviewed 
to  determine  if  any  significant 
design  changes  will  be  required  for 
manufacturing . 

6 

• 

14 

/ 

/ 

Test  program  results  and  the  status 
of  qualification  testing  to  determine 
production  impact  and  risk  have  been 
evaluated. 

7 

19 

Production  or  manufacturing  capabili- 
ties of  major  subcontractors  and  ven- 
dors have  been  technically  evaluated 
and  found  adequate. 

144 

Current 

Ranking  Question  AFSCR  84-2  PRR  Question 

Number 


8 


9 


10 


11 


12 


13 


14 


15 


9 Assurance  of  readiness  of  the  manu- 

facturing and  production  equipment, 
methods,  facilities,  test  and  train- 
ing equipment,  and  status  of  accessory 
and  ancillary  items. 

3 Critical  production  engineering  and 

production  tooling  have  been  demon- 
strated to  prove  that  engineering 
has  been  satisfactorily  accomplished. 

4*  Acquisition  will  smoothly  transition 

from  full-scale  development  to  pro- 
duction . 

10*  Planned  production  schedules  reflect 

economy  of  operations  and  minimize 
financial  commitments  until  all  major 
development  problems  have  been 
resolved. 

12  Change  activity  during  development 

has  been  evaluated  and  the  impact  of 
outstanding  changes  on  production  has 
been  assessed. 

15  Specifications  and  drawings  have  been 

reviewed  to  assure  their  adequacy  for 
the  planned  production  phase. 

11  A thorough  assessment  of  the  make-or- 

buy  structure  has  been  accomplished 
and  procedures  exist  so  control  and 
visibility  of  the  vendors  and  subcon- 
tractors can  be  effectively  maintained. 

21  Quality  controls  and  inspection  pro- 

cedures have  been  established  for 
materials  treatment  or  processes  to  be 
used  in  production. 


*Indicates  tied  pair  of  mean  rankings. 
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Ranking 

Current 

Question 

Number 

AFSCR  84-2  PRR  Question 

16 

25 

Planning  has  been  made  to  assure 
timely  release  of  manufacturing 
instructions . 

1 

17 

16 

Application  of  production  tooling 

and  test  equipment  to  manufacturing 

during  development  has  been  assessed 

and  the  application  of  same  to  the  1 

production  phase  has  been  defined. 

18 

18 

Production  management  systems  used 
for  providing  management  with  timely 
production  status  information  are 
effective. 

19 

17 

Material  management  system  for  deter- 
mination of  requirements,  procurement, 
receiving,  inspection,  materials 
handling  and  storage,  inventory  con- 
trol , control  of  finished  goods , and 
shipment  is  adequate. 

20 

7 

A systematic  approach  to  standardi- 
zation has  been  accomplished  in  the 
design  process  and  parts  selection 
to  maximize  the  use  of  military 
standard  components,  parts,  and  pro- 
cesses consistent  with  the  system 
requirements. 

% 

21 

20** 

Constraints  of  laboratory  or  model 
shop  capabilities  versus  quantity 
production  requirements  have  been 
fully  considered. 

22 

22** 

Assessment  of  the  GFP  or  services 
requirements,  controls,  management, 
and  availability  of  suppliers  has 
been  accomplished. 

♦♦Indicates 

tied  pair  of  mean  rankings. 

..  d 

- - - - . 
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Ranking 

Current 

Question 

Number 

AFSCR  84-2  PRR  Question 

23 

24 

The  contractor  is  adequately  organ- 
ized to  accomplish  the  production 
requirements . 

24 

23 

Availability  of  production  labor  skill 
requirements  has  been  assessed  and 
their  acquisition  adequately  planned. 

25 

8 

Product  assurance  controls  and 
tests  to  prevent  manufacturing  degra- 
dation of  performance  parameters 
have  been  established. 

I 
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